mirror of
https://github.com/status-im/nim-eth.git
synced 2025-01-11 14:54:33 +00:00
remove binary tries (#736)
Binary tries were an early research idea but are no longer part of any plausible etheruem roadmap, hence we remove them.
This commit is contained in:
Jacek Sieka
GitHub
parent
00f5fb1fe0
commit
b49df0a71a
329
doc/trie.md
329
doc/trie.md
@ -4,332 +4,3 @@ Nim Implementation of the Ethereum Trie structure
|
||||
|
||||
## Hexary Trie
|
||||
|
||||
## Binary Trie
|
||||
|
||||
Binary-trie is a dictionary-like data structure to store key-value pair.
|
||||
Much like it's sibling Hexary-trie, the key-value pair will be stored into key-value flat-db.
|
||||
The primary difference with Hexary-trie is, each node of Binary-trie only consist of one or two child,
|
||||
while Hexary-trie node can contains up to 16 or 17 child-nodes.
|
||||
|
||||
Unlike Hexary-trie, Binary-trie store it's data into flat-db without using rlp encoding.
|
||||
Binary-trie store its value using simple **Node-Types** encoding.
|
||||
The encoded-node will be hashed by keccak_256 and the hash value will be the key to flat-db.
|
||||
Each entry in the flat-db will looks like:
|
||||
|
||||
| key | value |
|
||||
|----------------------|--------------------------------------------|
|
||||
| 32-bytes-keccak-hash | encoded-node(KV or BRANCH or LEAF encoded) |
|
||||
|
||||
### Node-Types
|
||||
* KV = [0, encoded-key-path, 32 bytes hash of child]
|
||||
* BRANCH = [1, 32 bytes hash of left child, 32 bytes hash of right child]
|
||||
* LEAF = [2, value]
|
||||
|
||||
The KV node can have BRANCH node or LEAF node as it's child, but cannot a KV node.
|
||||
The internal algorithm will merge a KV(parent)->KV(child) into one KV node.
|
||||
Every KV node contains encoded keypath to reduce the number of blank nodes.
|
||||
|
||||
The BRANCH node can have KV, BRANCH, or LEAF node as it's children.
|
||||
|
||||
The LEAF node is the terminal node, it contains the value of a key.
|
||||
|
||||
### encoded-key-path
|
||||
|
||||
While Hexary-trie encode the path using Hex-Prefix encoding, Binary-trie
|
||||
encode the path using binary encoding, the scheme looks like this table below.
|
||||
|
||||
```text
|
||||
|--------- odd --------|
|
||||
00mm yyyy xxxx xxxx xxxx xxxx
|
||||
|------ even -----|
|
||||
1000 00mm yyyy xxxx xxxx xxxx
|
||||
```
|
||||
|
||||
| symbol | explanation |
|
||||
|--------|--------------------------|
|
||||
| xxxx | nibble of binary keypath in bits, 0 = left, 1 = right|
|
||||
| yyyy | nibble contains 0-3 bits padding + binary keypath |
|
||||
| mm | number of binary keypath bits modulo 4 (0-3) |
|
||||
| 00 | zero zero prefix |
|
||||
| 1000 | even numbered nibbles prefix |
|
||||
|
||||
if there is no padding, then yyyy bit sequence is absent, mm also zero.
|
||||
yyyy = mm bits + padding bits must be 4 bits length.
|
||||
|
||||
### The API
|
||||
|
||||
The primary API for Binary-trie is `set` and `get`.
|
||||
* set(key, value) --- _store a value associated with a key_
|
||||
* get(key): value --- _get a value using a key_
|
||||
|
||||
Both `key` and `value` are of `seq[byte]` type. And they cannot have zero length.
|
||||
|
||||
Getting a non-existent key will return zero length seq[byte].
|
||||
|
||||
Binary-trie also provide dictionary syntax API for `set` and `get`.
|
||||
* trie[key] = value -- same as `set`
|
||||
* value = trie[key] -- same as `get`
|
||||
* contains(key) a.k.a. `in` operator
|
||||
|
||||
Additional APIs are:
|
||||
* exists(key) -- returns `bool`, to check key-value existence -- same as contains
|
||||
* delete(key) -- remove a key-value from the trie
|
||||
* deleteSubtrie(key) -- remove a key-value from the trie plus all of it's subtrie
|
||||
that starts with the same key prefix
|
||||
* rootNode() -- get root node
|
||||
* rootNode(node) -- replace the root node
|
||||
* getRootHash(): `Hash32` with `seq[byte]` type
|
||||
* getDB(): `DB` -- get flat-db pointer
|
||||
|
||||
Constructor API:
|
||||
* initBinaryTrie(DB, rootHash[optional]) -- rootHash has `seq[byte]` or Hash32 type
|
||||
* init(BinaryTrie, DB, rootHash[optional])
|
||||
|
||||
Normally you would not set the rootHash when constructing an empty Binary-trie.
|
||||
Setting the rootHash occurred in a scenario where you have a populated DB
|
||||
with existing trie structure and you know the rootHash,
|
||||
and then you want to continue/resume the trie operations.
|
||||
|
||||
## Examples
|
||||
|
||||
```Nim
|
||||
import
|
||||
eth/trie/[db, binary, utils]
|
||||
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
trie.set("key1", "value1")
|
||||
trie.set("key2", "value2")
|
||||
doAssert trie.get("key1") == "value1".toBytes
|
||||
doAssert trie.get("key2") == "value2".toBytes
|
||||
|
||||
# delete all subtrie with key prefixes "key"
|
||||
trie.deleteSubtrie("key")
|
||||
doAssert trie.get("key1") == []
|
||||
doAssert trie.get("key2") == []]
|
||||
|
||||
trie["moon"] = "sun"
|
||||
doAssert "moon" in trie
|
||||
doAssert trie["moon"] == "sun".toBytes
|
||||
```
|
||||
|
||||
Remember, `set` and `get` are trie operations. A single `set` operation may invoke
|
||||
more than one store/lookup operation into the underlying DB. The same is also happened to `get` operation,
|
||||
it could do more than one flat-db lookup before it return the requested value.
|
||||
|
||||
## The truth behind a lie
|
||||
|
||||
What kind of lie? actually, `delete` and `deleteSubtrie` doesn't remove the
|
||||
'deleted' node from the underlying DB. It only make the node inaccessible
|
||||
from the user of the trie. The same also happened if you update the value of a key,
|
||||
the old value node is not removed from the underlying DB.
|
||||
A more subtle lie also happened when you add new entries into the trie using `set` operation.
|
||||
The previous hash of affected branch become obsolete and replaced by new hash,
|
||||
the old hash become inaccessible to the user.
|
||||
You may think that is a waste of storage space.
|
||||
Luckily, we also provide some utilities to deal with this situation, the branch utils.
|
||||
|
||||
## The branch utils
|
||||
|
||||
The branch utils consist of these API:
|
||||
* checkIfBranchExist(DB; rootHash; keyPrefix): bool
|
||||
* getBranch(DB; rootHash; key): branch
|
||||
* isValidBranch(branch, rootHash, key, value): bool
|
||||
* getWitness(DB; nodeHash; key): branch
|
||||
* getTrieNodes(DB; nodeHash): branch
|
||||
|
||||
`keyPrefix`, `key`, and `value` are bytes container with length greater than zero.
|
||||
They can be openArray[byte].
|
||||
|
||||
`rootHash` and `nodeHash` also bytes container,
|
||||
but they have constraint: must be 32 bytes in length, and it must be a keccak_256 hash value.
|
||||
|
||||
`branch` is a list of nodes, or in this case a `seq[seq[byte]]`.
|
||||
A list? yes, the structure is stored along with the encoded node.
|
||||
Therefore a list is enough to reconstruct the entire trie/branch.
|
||||
|
||||
```Nim
|
||||
import
|
||||
eth/trie/[db, binary, utils]
|
||||
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
trie.set("key1", "value1")
|
||||
trie.set("key2", "value2")
|
||||
|
||||
doAssert checkIfBranchExist(db, trie.getRootHash(), "key") == true
|
||||
doAssert checkIfBranchExist(db, trie.getRootHash(), "key1") == true
|
||||
doAssert checkIfBranchExist(db, trie.getRootHash(), "ken") == false
|
||||
doAssert checkIfBranchExist(db, trie.getRootHash(), "key123") == false
|
||||
```
|
||||
|
||||
The tree will looks like:
|
||||
```text
|
||||
root ---> A(kvnode, *common key prefix*)
|
||||
|
|
||||
|
|
||||
|
|
||||
B(branchnode)
|
||||
/ \
|
||||
/ \
|
||||
/ \
|
||||
C1(kvnode, *remain kepath*) C2(kvnode, *remain kepath*)
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
D1(leafnode, b'value1') D2(leafnode, b'value2')
|
||||
```
|
||||
|
||||
```Nim
|
||||
var branchA = getBranch(db, trie.getRootHash(), "key1")
|
||||
# ==> [A, B, C1, D1]
|
||||
|
||||
var branchB = getBranch(db, trie.getRootHash(), "key2")
|
||||
# ==> [A, B, C2, D2]
|
||||
|
||||
doAssert isValidBranch(branchA, trie.getRootHash(), "key1", "value1") == true
|
||||
# wrong key, return zero bytes
|
||||
doAssert isValidBranch(branchA, trie.getRootHash(), "key5", "") == true
|
||||
|
||||
doAssert isValidBranch(branchB, trie.getRootHash(), "key1", "value1") # InvalidNode
|
||||
|
||||
var x = getBranch(db, trie.getRootHash(), "key")
|
||||
# ==> [A]
|
||||
|
||||
x = getBranch(db, trie.getRootHash(), "key123") # InvalidKeyError
|
||||
x = getBranch(db, trie.getRootHash(), "key5") # there is still branch for non-exist key
|
||||
# ==> [A]
|
||||
|
||||
var branch = getWitness(db, trie.getRootHash(), "key1")
|
||||
# equivalent to `getBranch(db, trie.getRootHash(), "key1")`
|
||||
# ==> [A, B, C1, D1]
|
||||
|
||||
branch = getWitness(db, trie.getRootHash(), "key")
|
||||
# this will include additional nodes of "key2"
|
||||
# ==> [A, B, C1, D1, C2, D2]
|
||||
|
||||
var wholeTrie = getWitness(db, trie.getRootHash(), "")
|
||||
# this will return the whole trie
|
||||
# ==> [A, B, C1, D1, C2, D2]
|
||||
|
||||
var node = branch[1] # B
|
||||
let nodeHash = keccak256.digest(node.baseAddr, uint(node.len))
|
||||
var nodes = getTrieNodes(db, nodeHash)
|
||||
doAssert nodes.len == wholeTrie.len - 1
|
||||
# ==> [B, C1, D1, C2, D2]
|
||||
```
|
||||
|
||||
## Remember the lie?
|
||||
|
||||
Because trie `delete`, `deleteSubtrie` and `set` operation create inaccessible nodes in the underlying DB,
|
||||
we need to remove them if necessary. We already see that `wholeTrie = getWitness(db, trie.getRootHash(), "")`
|
||||
will return the whole trie, a list of accessible nodes.
|
||||
Then we can write the clean tree into a new DB instance to replace the old one.
|
||||
|
||||
|
||||
## Sparse Merkle Trie
|
||||
|
||||
Sparse Merkle Trie(SMT) is a variant of Binary Trie which uses binary encoding to
|
||||
represent path during trie traversal. When Binary Trie uses three types of node,
|
||||
SMT only use one type of node without any additional special encoding to store it's key-path.
|
||||
|
||||
Actually, it doesn't even store it's key-path anywhere like Binary Trie,
|
||||
the key-path is stored implicitly in the trie structure during key-value insertion.
|
||||
|
||||
Because the key-path is not encoded in any special ways, the bits can be extracted directly from
|
||||
the key without any conversion.
|
||||
|
||||
However, the key restricted to a fixed length because the algorithm demand a fixed height trie
|
||||
to works properly. In this case, the trie height is limited to 160 level,
|
||||
or the key is of fixed length 20 bytes (8 bits x 20 = 160).
|
||||
|
||||
To be able to use variable length key, the algorithm can be adapted slightly using hashed key before
|
||||
constructing the binary key-path. For example, if using keccak256 as the hashing function,
|
||||
then the height of the tree will be 256, but the key itself can be any length.
|
||||
|
||||
### The API
|
||||
|
||||
The primary API for Binary-trie is `set` and `get`.
|
||||
* set(key, value, rootHash[optional]) --- _store a value associated with a key_
|
||||
* get(key, rootHash[optional]): value --- _get a value using a key_
|
||||
|
||||
Both `key` and `value` are of `BytesRange` type. And they cannot have zero length.
|
||||
You can also use convenience API `get` and `set` which accepts
|
||||
`Bytes` or `string` (a `string` is conceptually wrong in this context
|
||||
and may costlier than a `BytesRange`, but it is good for testing purpose).
|
||||
|
||||
rootHash is an optional parameter. When used, `get` will get a key from specific root,
|
||||
and `set` will also set a key at specific root.
|
||||
|
||||
Getting a non-existent key will return zero length BytesRange or a zeroBytesRange.
|
||||
|
||||
Sparse Merkle Trie also provide dictionary syntax API for `set` and `get`.
|
||||
* trie[key] = value -- same as `set`
|
||||
* value = trie[key] -- same as `get`
|
||||
* contains(key) a.k.a. `in` operator
|
||||
|
||||
Additional APIs are:
|
||||
* exists(key) -- returns `bool`, to check key-value existence -- same as contains
|
||||
* delete(key) -- remove a key-value from the trie
|
||||
* getRootHash(): `KeccakHash` with `BytesRange` type
|
||||
* getDB(): `DB` -- get flat-db pointer
|
||||
* prove(key, rootHash[optional]): proof -- useful for merkling
|
||||
|
||||
Constructor API:
|
||||
* initSparseBinaryTrie(DB, rootHash[optional])
|
||||
* init(SparseBinaryTrie, DB, rootHash[optional])
|
||||
|
||||
Normally you would not set the rootHash when constructing an empty Sparse Merkle Trie.
|
||||
Setting the rootHash occurred in a scenario where you have a populated DB
|
||||
with existing trie structure and you know the rootHash,
|
||||
and then you want to continue/resume the trie operations.
|
||||
|
||||
## Examples
|
||||
|
||||
```Nim
|
||||
import
|
||||
eth/trie/[db, sparse_binary, utils]
|
||||
|
||||
var
|
||||
db = newMemoryDB()
|
||||
trie = initSparseMerkleTrie(db)
|
||||
|
||||
let
|
||||
key1 = "01234567890123456789"
|
||||
key2 = "abcdefghijklmnopqrst"
|
||||
|
||||
trie.set(key1, "value1")
|
||||
trie.set(key2, "value2")
|
||||
doAssert trie.get(key1) == "value1".toBytes
|
||||
doAssert trie.get(key2) == "value2".toBytes
|
||||
|
||||
trie.delete(key1)
|
||||
doAssert trie.get(key1) == []
|
||||
|
||||
trie.delete(key2)
|
||||
doAssert trie[key2] == []
|
||||
```
|
||||
|
||||
Remember, `set` and `get` are trie operations. A single `set` operation may invoke
|
||||
more than one store/lookup operation into the underlying DB. The same is also happened to `get` operation,
|
||||
it could do more than one flat-db lookup before it return the requested value.
|
||||
While Binary Trie perform a variable numbers of lookup and store operations, Sparse Merkle Trie
|
||||
will do constant numbers of lookup and store operations each `get` and `set` operation.
|
||||
|
||||
## Merkle Proofing
|
||||
|
||||
Using ``prove`` dan ``verifyProof`` API, we can do some merkling with SMT.
|
||||
|
||||
```Nim
|
||||
let
|
||||
value1 = "hello world"
|
||||
badValue = "bad value"
|
||||
|
||||
trie[key1] = value1
|
||||
var proof = trie.prove(key1)
|
||||
|
||||
doAssert verifyProof(proof, trie.getRootHash(), key1, value1) == true
|
||||
doAssert verifyProof(proof, trie.getRootHash(), key1, badValue) == false
|
||||
doAssert verifyProof(proof, trie.getRootHash(), key2, value1) == false
|
||||
```
|
||||
|
||||
|
||||
@ -1,143 +0,0 @@
|
||||
import
|
||||
std/sequtils,
|
||||
stew/ptrops,
|
||||
"."/[trie_defs, trie_bitseq]
|
||||
|
||||
type
|
||||
TrieNodeKind* = enum
|
||||
KV_TYPE = 0
|
||||
BRANCH_TYPE = 1
|
||||
LEAF_TYPE = 2
|
||||
|
||||
TrieNodeKey* = seq[byte]
|
||||
|
||||
TrieNode* = object
|
||||
case kind*: TrieNodeKind
|
||||
of KV_TYPE:
|
||||
keyPath*: TrieBitSeq
|
||||
child*: TrieNodeKey
|
||||
of BRANCH_TYPE:
|
||||
leftChild*: TrieNodeKey
|
||||
rightChild*: TrieNodeKey
|
||||
of LEAF_TYPE:
|
||||
value*: seq[byte]
|
||||
|
||||
InvalidNode* = object of CorruptedTrieDatabase
|
||||
ValidationError* = object of CorruptedTrieDatabase
|
||||
|
||||
# ----------------------------------------------
|
||||
template sliceToEnd*(r: TrieBitSeq, index: int): TrieBitSeq =
|
||||
if r.len <= index: TrieBitSeq() else: r[index .. ^1]
|
||||
|
||||
proc decodeToBinKeypath*(path: seq[byte]): TrieBitSeq =
|
||||
## Decodes bytes into a sequence of 0s and 1s
|
||||
## Used in decoding key path of a KV-NODE
|
||||
var path = path.bits
|
||||
if path[0]:
|
||||
path = path[4..^1]
|
||||
|
||||
doAssert path[0] == false
|
||||
doAssert path[1] == false
|
||||
var bits = path[2].int shl 1
|
||||
bits = bits or path[3].int
|
||||
|
||||
if path.len > 4:
|
||||
path[4+((4 - bits) mod 4)..^1]
|
||||
else:
|
||||
TrieBitSeq()
|
||||
|
||||
proc parseNode*(node: openArray[byte]): TrieNode =
|
||||
# Input: a serialized node
|
||||
|
||||
if node.len == 0:
|
||||
raise newException(InvalidNode, "Blank node is not a valid node type in Binary Trie")
|
||||
|
||||
if node[0].ord < low(TrieNodeKind).ord or node[0].ord > high(TrieNodeKind).ord:
|
||||
raise newException(InvalidNode, "Invalid node type")
|
||||
|
||||
let nodeType = node[0].TrieNodeKind
|
||||
case nodeType
|
||||
of BRANCH_TYPE:
|
||||
if node.len != 65:
|
||||
raise newException(InvalidNode, "Invalid branch node, both child node should be 32 bytes long each")
|
||||
# Output: node type, left child, right child
|
||||
result = TrieNode(kind: BRANCH_TYPE, leftChild: node[1..<33], rightChild: node[33..^1])
|
||||
doAssert(result.leftChild.len == 32)
|
||||
doAssert(result.rightChild.len == 32)
|
||||
return result
|
||||
of KV_TYPE:
|
||||
if node.len <= 33:
|
||||
raise newException(InvalidNode, "Invalid kv node, short of key path or child node hash")
|
||||
# Output: node type, keypath, child
|
||||
return TrieNode(kind: KV_TYPE, keyPath: decodeToBinKeypath(node[1..^33]), child: node[^32..^1])
|
||||
of LEAF_TYPE:
|
||||
if node.len == 1:
|
||||
raise newException(InvalidNode, "Invalid leaf node, can not contain empty value")
|
||||
# Output: node type, value
|
||||
return TrieNode(kind: LEAF_TYPE, value: node[1..^1])
|
||||
|
||||
proc encodeKVNode*(keyPath: TrieBitSeq, childHash: TrieNodeKey): seq[byte] =
|
||||
## Serializes a key/value node
|
||||
if keyPath.len == 0:
|
||||
raise newException(ValidationError, "Key path can not be empty")
|
||||
|
||||
if childHash.len != 32:
|
||||
raise newException(ValidationError, "Invalid hash len")
|
||||
|
||||
# Encodes a sequence of 0s and 1s into tightly packed bytes
|
||||
# Used in encoding key path of a KV-NODE
|
||||
# KV-NODE = KV-TYPE-PREFIX + encoded keypath + 32 bytes hash
|
||||
let
|
||||
len = keyPath.len
|
||||
padding = ((not len) + 1) and 3 # modulo 4 padding
|
||||
paddedBinLen = len + padding
|
||||
prefix = len mod 4
|
||||
|
||||
result = newSeq[byte](((len + padding) div 8) + 34)
|
||||
result[0] = KV_TYPE.byte
|
||||
if paddedBinLen mod 8 == 4:
|
||||
var nbits = 4 - padding
|
||||
result[1] = byte(prefix shl 4) or byte.fromBits(keyPath, 0, nbits)
|
||||
for i in 0..<(len div 8):
|
||||
result[i+2] = byte.fromBits(keyPath, nbits, 8)
|
||||
inc(nbits, 8)
|
||||
else:
|
||||
var nbits = 8 - padding
|
||||
result[1] = byte(0b1000_0000) or byte(prefix)
|
||||
result[2] = byte.fromBits(keyPath, 0, nbits)
|
||||
for i in 0..<((len-1) div 8):
|
||||
result[i+3] = byte.fromBits(keyPath, nbits, 8)
|
||||
inc(nbits, 8)
|
||||
copyMem(result[^32].addr, childHash.baseAddr, 32)
|
||||
|
||||
proc encodeKVNode*(keyPath: bool, childHash: TrieNodeKey): seq[byte] =
|
||||
result = newSeq[byte](34)
|
||||
result[0] = KV_TYPE.byte
|
||||
result[1] = byte(16) or byte(keyPath)
|
||||
copyMem(result[^32].addr, childHash.baseAddr, 32)
|
||||
|
||||
proc encodeBranchNode*(leftChildHash, rightChildHash: TrieNodeKey): seq[byte] =
|
||||
## Serializes a branch node
|
||||
const
|
||||
BRANCH_TYPE_PREFIX = @[BRANCH_TYPE.byte]
|
||||
|
||||
if leftChildHash.len != 32 or rightChildHash.len != 32:
|
||||
raise newException(ValidationError, "encodeBranchNode: Invalid hash len")
|
||||
|
||||
result = BRANCH_TYPE_PREFIX.concat(leftChildHash, rightChildHash)
|
||||
|
||||
proc encodeLeafNode*(value: openArray[byte]): seq[byte] =
|
||||
## Serializes a leaf node
|
||||
const
|
||||
LEAF_TYPE_PREFIX = @[LEAF_TYPE.byte]
|
||||
|
||||
if value.len == 0:
|
||||
raise newException(ValidationError, "Value of leaf node can not be empty")
|
||||
|
||||
result = LEAF_TYPE_PREFIX.concat(@value)
|
||||
|
||||
proc getCommonPrefixLength*(a, b: TrieBitSeq): int =
|
||||
let len = min(a.len, b.len)
|
||||
for i in 0..<len:
|
||||
if a[i] != b[i]: return i
|
||||
result = len
|
||||
@ -1,284 +0,0 @@
|
||||
import
|
||||
"."/[trie_bitseq, trie_defs, db, binaries, trie_utils]
|
||||
|
||||
export
|
||||
trie_utils
|
||||
|
||||
type
|
||||
DB = TrieDatabaseRef
|
||||
|
||||
BinaryTrie* = object
|
||||
db: DB
|
||||
rootHash: TrieNodeKey
|
||||
|
||||
NodeOverrideError* = object of CatchableError
|
||||
|
||||
const
|
||||
zeroHash* = default(seq[byte])
|
||||
|
||||
proc init*(x: typedesc[BinaryTrie], db: DB,
|
||||
rootHash: openArray[byte]): BinaryTrie =
|
||||
checkValidHashZ(rootHash)
|
||||
result.db = db
|
||||
result.rootHash = @(rootHash)
|
||||
|
||||
proc getDB*(t: BinaryTrie): auto = t.db
|
||||
|
||||
proc initBinaryTrie*(db: DB, rootHash: openArray[byte]): BinaryTrie =
|
||||
init(BinaryTrie, db, rootHash)
|
||||
|
||||
proc initBinaryTrie*(db: DB): BinaryTrie =
|
||||
init(BinaryTrie, db, zeroHash)
|
||||
|
||||
proc getRootHash*(self: BinaryTrie): TrieNodeKey {.inline.} =
|
||||
self.rootHash
|
||||
|
||||
template fetchNode(self: BinaryTrie, nodeHash: TrieNodeKey): TrieNode =
|
||||
doAssert(nodeHash.len == 32)
|
||||
parseNode self.db.get(nodeHash)
|
||||
|
||||
proc getAux(self: BinaryTrie, nodeHash: TrieNodeKey, keyPath: TrieBitSeq): seq[byte] =
|
||||
# Empty trie
|
||||
if isZeroHash(nodeHash):
|
||||
return
|
||||
|
||||
let node = self.fetchNode(nodeHash)
|
||||
|
||||
# Key-value node descend
|
||||
if node.kind == LEAF_TYPE:
|
||||
if keyPath.len != 0: return
|
||||
return node.value
|
||||
elif node.kind == KV_TYPE:
|
||||
# keyPath too short
|
||||
if keyPath.len == 0: return
|
||||
let sliceLen = min(node.keyPath.len, keyPath.len)
|
||||
if keyPath[0..<sliceLen] == node.keyPath:
|
||||
return self.getAux(node.child, keyPath.sliceToEnd(node.keyPath.len))
|
||||
else:
|
||||
return
|
||||
# Branch node descend
|
||||
elif node.kind == BRANCH_TYPE:
|
||||
# keyPath too short
|
||||
if keyPath.len == 0: return
|
||||
if keyPath[0]: # first bit == 1
|
||||
return self.getAux(node.rightChild, keyPath.sliceToEnd(1))
|
||||
else:
|
||||
return self.getAux(node.leftChild, keyPath.sliceToEnd(1))
|
||||
|
||||
proc get*(self: BinaryTrie, key: openArray[byte]): seq[byte] {.inline.} =
|
||||
var keyBits = key.bits
|
||||
return self.getAux(self.rootHash, keyBits)
|
||||
|
||||
proc hashAndSave*(self: BinaryTrie, node: openArray[byte]): TrieNodeKey =
|
||||
result = @(keccak256(node).data)
|
||||
self.db.put(result, node)
|
||||
|
||||
template saveKV(self: BinaryTrie, keyPath: TrieBitSeq | bool, child: openArray[byte]): untyped =
|
||||
self.hashAndSave(encodeKVNode(keyPath, child))
|
||||
|
||||
template saveLeaf(self: BinaryTrie, value: openArray[byte]): untyped =
|
||||
self.hashAndSave(encodeLeafNode(value))
|
||||
|
||||
template saveBranch(self: BinaryTrie, L, R: openArray[byte]): untyped =
|
||||
self.hashAndSave(encodeBranchNode(L, R))
|
||||
|
||||
proc setBranchNode(self: BinaryTrie, keyPath: TrieBitSeq, node: TrieNode,
|
||||
value: openArray[byte], deleteSubtrie = false): TrieNodeKey
|
||||
proc setKVNode(self: BinaryTrie, keyPath: TrieBitSeq, nodeHash: TrieNodeKey,
|
||||
node: TrieNode, value: openArray[byte], deleteSubtrie = false): TrieNodeKey
|
||||
|
||||
const
|
||||
overrideErrorMsg =
|
||||
"Fail to set the value because the prefix of it's key is the same as existing key"
|
||||
|
||||
proc setAux(self: BinaryTrie, nodeHash: TrieNodeKey, keyPath: TrieBitSeq,
|
||||
value: openArray[byte], deleteSubtrie = false): TrieNodeKey =
|
||||
## If deleteSubtrie is set to True, what it will do is that it take in a keyPath
|
||||
## and traverse til the end of keyPath, then delete the whole subtrie of that node.
|
||||
## Note: keyPath should be in binary array format, i.e., encoded by encode_to_bin()
|
||||
|
||||
template checkBadKeyPath(): untyped =
|
||||
# keyPath too short
|
||||
if keyPath.len == 0:
|
||||
if deleteSubtrie: return zeroHash
|
||||
else: raise newException(NodeOverrideError, overrideErrorMsg)
|
||||
|
||||
template ifGoodValue(body: untyped): untyped =
|
||||
if value.len != 0: body
|
||||
else: return zeroHash
|
||||
|
||||
# Empty trie
|
||||
if isZeroHash(nodeHash):
|
||||
ifGoodValue:
|
||||
return self.saveKV(keyPath, self.saveLeaf(value))
|
||||
|
||||
let node = self.fetchNode(nodeHash)
|
||||
|
||||
case node.kind
|
||||
of LEAF_TYPE: # Node is a leaf node
|
||||
# keyPath must match, there should be no remaining keyPath
|
||||
if keyPath.len != 0:
|
||||
raise newException(NodeOverrideError, overrideErrorMsg)
|
||||
if deleteSubtrie: return zeroHash
|
||||
|
||||
ifGoodValue:
|
||||
return self.saveLeaf(value)
|
||||
of KV_TYPE: # node is a key-value node
|
||||
checkBadKeyPath()
|
||||
return self.setKVNode(keyPath, nodeHash, node, value, deleteSubtrie)
|
||||
of BRANCH_TYPE: # node is a branch node
|
||||
checkBadKeyPath()
|
||||
return self.setBranchNode(keyPath, node, value, deleteSubtrie)
|
||||
|
||||
proc set*(self: var BinaryTrie, key, value: openArray[byte]) {.inline.} =
|
||||
## Sets the value at the given keyPath from the given node
|
||||
## Key will be encoded into binary array format first.
|
||||
|
||||
var keyBits = key.bits
|
||||
self.rootHash = self.setAux(self.rootHash, keyBits, value)
|
||||
|
||||
proc setBranchNode(self: BinaryTrie, keyPath: TrieBitSeq, node: TrieNode,
|
||||
value: openArray[byte], deleteSubtrie = false): TrieNodeKey =
|
||||
# Which child node to update? Depends on first bit in keyPath
|
||||
var newLeftChild, newRightChild: TrieNodeKey
|
||||
|
||||
if keyPath[0]: # first bit == 1
|
||||
newRightChild = self.setAux(node.rightChild, keyPath[1..^1], value, deleteSubtrie)
|
||||
newLeftChild = node.leftChild
|
||||
else:
|
||||
newLeftChild = self.setAux(node.leftChild, keyPath[1..^1], value, deleteSubtrie)
|
||||
newRightChild = node.rightChild
|
||||
|
||||
let blankLeft = isZeroHash(newLeftChild)
|
||||
|
||||
# Compress branch node into kv node
|
||||
if blankLeft or isZeroHash(newRightChild):
|
||||
let childNode = if blankLeft: newRightChild else: newLeftChild
|
||||
var subNode = self.fetchNode(childNode)
|
||||
|
||||
# Compress (k1, (k2, NODE)) -> (k1 + k2, NODE)
|
||||
if subNode.kind == KV_TYPE:
|
||||
# exploit subNode.keyPath unused prefix bit
|
||||
# to avoid bitVector concat
|
||||
subNode.keyPath.pushFront(blankLeft)
|
||||
result = self.saveKV(subNode.keyPath, subNode.child)
|
||||
# kv node pointing to a branch node
|
||||
elif subNode.kind in {BRANCH_TYPE, LEAF_TYPE}:
|
||||
result = self.saveKV(blankLeft, childNode)
|
||||
else:
|
||||
result = self.saveBranch(newLeftChild, newRightChild)
|
||||
|
||||
proc setKVNode(self: BinaryTrie, keyPath: TrieBitSeq, nodeHash: TrieNodeKey,
|
||||
node: TrieNode, value: openArray[byte], deleteSubtrie = false): TrieNodeKey =
|
||||
# keyPath prefixes match
|
||||
if deleteSubtrie:
|
||||
if keyPath.len < node.keyPath.len and keyPath == node.keyPath[0..<keyPath.len]:
|
||||
return zeroHash
|
||||
|
||||
let sliceLen = min(node.keyPath.len, keyPath.len)
|
||||
|
||||
if keyPath[0..<sliceLen] == node.keyPath:
|
||||
# Recurse into child
|
||||
let subNodeHash = self.setAux(node.child,
|
||||
keyPath.sliceToEnd(node.keyPath.len), value, deleteSubtrie)
|
||||
|
||||
# If child is empty
|
||||
if isZeroHash(subNodeHash):
|
||||
return zeroHash
|
||||
let subNode = self.fetchNode(subNodeHash)
|
||||
|
||||
# If the child is a key-value node, compress together the keyPaths
|
||||
# into one node
|
||||
if subNode.kind == KV_TYPE:
|
||||
return self.saveKV(node.keyPath & subNode.keyPath, subNode.child)
|
||||
else:
|
||||
return self.saveKV(node.keyPath, subNodeHash)
|
||||
# keyPath prefixes don't match. Here we will be converting a key-value node
|
||||
# of the form (k, CHILD) into a structure of one of the following forms:
|
||||
# 1. (k[:-1], (NEWCHILD, CHILD))
|
||||
# 2. (k[:-1], ((k2, NEWCHILD), CHILD))
|
||||
# 3. (k1, ((k2, CHILD), NEWCHILD))
|
||||
# 4. (k1, ((k2, CHILD), (k2', NEWCHILD))
|
||||
# 5. (CHILD, NEWCHILD)
|
||||
# 6. ((k[1:], CHILD), (k', NEWCHILD))
|
||||
# 7. ((k[1:], CHILD), NEWCHILD)
|
||||
# 8. (CHILD, (k[1:], NEWCHILD))
|
||||
else:
|
||||
let
|
||||
commonPrefixLen = getCommonPrefixLength(node.keyPath, keyPath[0..<sliceLen])
|
||||
cplenPlusOne = commonPrefixLen + 1
|
||||
# New key-value pair can not contain empty value
|
||||
# Or one can not delete non-exist subtrie
|
||||
if value.len == 0 or deleteSubtrie: return nodeHash
|
||||
|
||||
var valNode, oldNode, newSub: TrieNodeKey
|
||||
# valnode: the child node that has the new value we are adding
|
||||
# Case 1: keyPath prefixes almost match, so we are in case (1), (2), (5), (6)
|
||||
if keyPath.len == cplenPlusOne:
|
||||
valNode = self.saveLeaf(value)
|
||||
# Case 2: keyPath prefixes mismatch in the middle, so we need to break
|
||||
# the keyPath in half. We are in case (3), (4), (7), (8)
|
||||
else:
|
||||
if keyPath.len <= commonPrefixLen:
|
||||
raise newException(NodeOverrideError, overrideErrorMsg)
|
||||
valNode = self.saveKV(keyPath[cplenPlusOne..^1], self.saveLeaf(value))
|
||||
|
||||
# oldnode: the child node the has the old child value
|
||||
# Case 1: (1), (3), (5), (6)
|
||||
if node.keyPath.len == cplenPlusOne:
|
||||
oldNode = node.child
|
||||
# (2), (4), (6), (8)
|
||||
else:
|
||||
oldNode = self.saveKV(node.keyPath[cplenPlusOne..^1], node.child)
|
||||
|
||||
# Create the new branch node (because the key paths diverge, there has to
|
||||
# be some "first bit" at which they diverge, so there must be a branch
|
||||
# node somewhere)
|
||||
if keyPath[commonPrefixLen]: # first bit == 1
|
||||
newSub = self.saveBranch(oldNode, valNode)
|
||||
else:
|
||||
newSub = self.saveBranch(valNode, oldNode)
|
||||
|
||||
# Case 1: keyPath prefixes match in the first bit, so we still need
|
||||
# a kv node at the top
|
||||
# (1) (2) (3) (4)
|
||||
if commonPrefixLen != 0:
|
||||
return self.saveKV(node.keyPath[0..<commonPrefixLen], newSub)
|
||||
# Case 2: keyPath prefixes diverge in the first bit, so we replace the
|
||||
# kv node with a branch node
|
||||
# (5) (6) (7) (8)
|
||||
else:
|
||||
return newSub
|
||||
|
||||
template exists*(self: BinaryTrie, key: openArray[byte]): bool =
|
||||
self.get(key) != []
|
||||
|
||||
proc delete*(self: var BinaryTrie, key: openArray[byte]) {.inline.} =
|
||||
## Equals to setting the value to zeroBytesRange
|
||||
var keyBits = key.bits
|
||||
self.rootHash = self.setAux(self.rootHash, keyBits, [])
|
||||
|
||||
proc deleteSubtrie*(self: var BinaryTrie, key: openArray[byte]) {.inline.} =
|
||||
## Given a key prefix, delete the whole subtrie that starts with the key prefix.
|
||||
## Key will be encoded into binary array format first.
|
||||
## It will call `setAux` with `deleteSubtrie` set to true.
|
||||
|
||||
var keyBits = key.bits
|
||||
self.rootHash = self.setAux(self.rootHash, keyBits, [], true)
|
||||
|
||||
# Convenience
|
||||
proc rootNode*(self: BinaryTrie): seq[byte] {.inline.} =
|
||||
self.db.get(self.rootHash)
|
||||
|
||||
proc rootNode*(self: var BinaryTrie, node: openArray[byte]) {.inline.} =
|
||||
self.rootHash = self.hashAndSave(node)
|
||||
|
||||
# Dictionary API
|
||||
template `[]`*(self: BinaryTrie, key: seq[byte]): seq[byte] =
|
||||
self.get(key)
|
||||
|
||||
template `[]=`*(self: var BinaryTrie, key, value: seq[byte]) =
|
||||
self.set(key, value)
|
||||
|
||||
template contains*(self: BinaryTrie, key: seq[byte]): bool =
|
||||
self.exists(key)
|
||||
@ -1,168 +0,0 @@
|
||||
import
|
||||
"."/[trie_defs, binary, binaries, db, trie_utils, trie_bitseq]
|
||||
|
||||
type
|
||||
DB = TrieDatabaseRef
|
||||
|
||||
# TODO: replace the usages of this with regular asserts
|
||||
InvalidKeyError* = object of Defect
|
||||
|
||||
template query(db: DB, nodeHash: TrieNodeKey): seq[byte] =
|
||||
db.get(nodeHash)
|
||||
|
||||
proc checkIfBranchExistImpl(db: DB; nodeHash: TrieNodeKey; keyPrefix: TrieBitSeq): bool =
|
||||
if nodeHash == zeroHash:
|
||||
return false
|
||||
|
||||
let node = parseNode(db.query(nodeHash))
|
||||
|
||||
case node.kind:
|
||||
of LEAF_TYPE:
|
||||
if keyPrefix.len != 0: return false
|
||||
return true
|
||||
of KV_TYPE:
|
||||
if keyPrefix.len == 0: return true
|
||||
if keyPrefix.len < node.keyPath.len:
|
||||
if keyPrefix == node.keyPath[0..<keyPrefix.len]: return true
|
||||
return false
|
||||
else:
|
||||
if keyPrefix[0..<node.keyPath.len] == node.keyPath:
|
||||
return checkIfBranchExistImpl(db, node.child, keyPrefix.sliceToEnd(node.keyPath.len))
|
||||
return false
|
||||
of BRANCH_TYPE:
|
||||
if keyPrefix.len == 0: return true
|
||||
if keyPrefix[0] == false:
|
||||
return checkIfBranchExistImpl(db, node.leftChild, keyPrefix.sliceToEnd(1))
|
||||
else:
|
||||
return checkIfBranchExistImpl(db, node.rightChild, keyPrefix.sliceToEnd(1))
|
||||
|
||||
proc checkIfBranchExist*(db: DB; rootHash: TrieNodeKey, keyPrefix: openArray[byte]): bool =
|
||||
## Given a key prefix, return whether this prefix is
|
||||
## the prefix of an existing key in the trie.
|
||||
checkValidHashZ(rootHash)
|
||||
var keyPrefixBits = bits keyPrefix
|
||||
checkIfBranchExistImpl(db, rootHash, keyPrefixBits)
|
||||
|
||||
proc getBranchImpl(db: DB; nodeHash: TrieNodeKey, keyPath: TrieBitSeq, output: var seq[seq[byte]]) =
|
||||
if nodeHash == zeroHash: return
|
||||
|
||||
let nodeVal = db.query(nodeHash)
|
||||
let node = parseNode(nodeVal)
|
||||
|
||||
case node.kind
|
||||
of LEAF_TYPE:
|
||||
if keyPath.len == 0:
|
||||
output.add nodeVal
|
||||
else:
|
||||
raise newException(InvalidKeyError, "Key too long")
|
||||
|
||||
of KV_TYPE:
|
||||
if keyPath.len == 0:
|
||||
raise newException(InvalidKeyError, "Key too short")
|
||||
|
||||
output.add nodeVal
|
||||
let sliceLen = min(keyPath.len, node.keyPath.len)
|
||||
if keyPath[0..<sliceLen] == node.keyPath:
|
||||
getBranchImpl(db, node.child, keyPath.sliceToEnd(sliceLen), output)
|
||||
|
||||
of BRANCH_TYPE:
|
||||
if keyPath.len == 0:
|
||||
raise newException(InvalidKeyError, "Key too short")
|
||||
|
||||
output.add nodeVal
|
||||
if keyPath[0] == false:
|
||||
getBranchImpl(db, node.leftChild, keyPath.sliceToEnd(1), output)
|
||||
else:
|
||||
getBranchImpl(db, node.rightChild, keyPath.sliceToEnd(1), output)
|
||||
|
||||
proc getBranch*(db: DB; rootHash: seq[byte]; key: openArray[byte]): seq[seq[byte]] =
|
||||
## Get a long-format Merkle branch
|
||||
checkValidHashZ(rootHash)
|
||||
result = @[]
|
||||
var keyBits = bits key
|
||||
getBranchImpl(db, rootHash, keyBits, result)
|
||||
|
||||
proc isValidBranch*(branch: seq[seq[byte]], rootHash: seq[byte], key, value: openArray[byte]): bool =
|
||||
checkValidHashZ(rootHash)
|
||||
# branch must not be empty
|
||||
doAssert(branch.len != 0)
|
||||
|
||||
var db = newMemoryDB()
|
||||
for node in branch:
|
||||
doAssert(node.len != 0)
|
||||
let nodeHash = keccak256(node)
|
||||
db.put(nodeHash.data, node)
|
||||
|
||||
var trie = initBinaryTrie(db, rootHash)
|
||||
result = trie.get(key) == value
|
||||
|
||||
proc getTrieNodesImpl(db: DB; nodeHash: TrieNodeKey, output: var seq[seq[byte]]): bool =
|
||||
## Get full trie of a given root node
|
||||
|
||||
if nodeHash.isZeroHash(): return false
|
||||
|
||||
var nodeVal: seq[byte]
|
||||
if nodeHash in db:
|
||||
nodeVal = db.query(nodeHash)
|
||||
else:
|
||||
return false
|
||||
|
||||
let node = parseNode(nodeVal)
|
||||
|
||||
case node.kind
|
||||
of KV_TYPE:
|
||||
output.add nodeVal
|
||||
result = getTrieNodesImpl(db, node.child, output)
|
||||
of BRANCH_TYPE:
|
||||
output.add nodeVal
|
||||
result = getTrieNodesImpl(db, node.leftChild, output)
|
||||
result = getTrieNodesImpl(db, node.rightChild, output)
|
||||
of LEAF_TYPE:
|
||||
output.add nodeVal
|
||||
|
||||
proc getTrieNodes*(db: DB; nodeHash: TrieNodeKey): seq[seq[byte]] =
|
||||
checkValidHashZ(nodeHash)
|
||||
result = @[]
|
||||
discard getTrieNodesImpl(db, nodeHash, result)
|
||||
|
||||
proc getWitnessImpl*(db: DB; nodeHash: TrieNodeKey; keyPath: TrieBitSeq; output: var seq[seq[byte]]) =
|
||||
if keyPath.len == 0:
|
||||
if not getTrieNodesImpl(db, nodeHash, output): return
|
||||
|
||||
if nodeHash.isZeroHash(): return
|
||||
|
||||
var nodeVal: seq[byte]
|
||||
if nodeHash in db:
|
||||
nodeVal = db.query(nodeHash)
|
||||
else:
|
||||
return
|
||||
|
||||
let node = parseNode(nodeVal)
|
||||
|
||||
case node.kind
|
||||
of LEAF_TYPE:
|
||||
if keyPath.len != 0:
|
||||
raise newException(InvalidKeyError, "Key too long")
|
||||
of KV_TYPE:
|
||||
output.add nodeVal
|
||||
if keyPath.len < node.keyPath.len and node.keyPath[0..<keyPath.len] == keyPath:
|
||||
if not getTrieNodesImpl(db, node.child, output): return
|
||||
elif keyPath[0..<node.keyPath.len] == node.keyPath:
|
||||
getWitnessImpl(db, node.child, keyPath.sliceToEnd(node.keyPath.len), output)
|
||||
of BRANCH_TYPE:
|
||||
output.add nodeVal
|
||||
if keyPath[0] == false:
|
||||
getWitnessImpl(db, node.leftChild, keyPath.sliceToEnd(1), output)
|
||||
else:
|
||||
getWitnessImpl(db, node.rightChild, keyPath.sliceToEnd(1), output)
|
||||
|
||||
proc getWitness*(db: DB; nodeHash: TrieNodeKey; key: openArray[byte]): seq[seq[byte]] =
|
||||
## Get all witness given a keyPath prefix.
|
||||
## Include
|
||||
##
|
||||
## 1. witness along the keyPath and
|
||||
## 2. witness in the subtrie of the last node in keyPath
|
||||
checkValidHashZ(nodeHash)
|
||||
result = @[]
|
||||
var keyBits = bits key
|
||||
getWitnessImpl(db, nodeHash, keyBits, result)
|
||||
@ -1,129 +0,0 @@
|
||||
import
|
||||
stew/bitops2
|
||||
|
||||
type
|
||||
TrieBitSeq* = object
|
||||
## Bit sequence as used in ethereum tries
|
||||
data: seq[byte]
|
||||
start: int
|
||||
mLen: int ## Length in bits
|
||||
|
||||
template `@`(s, idx: untyped): untyped =
|
||||
(when idx is BackwardsIndex: s.len - int(idx) else: int(idx))
|
||||
|
||||
proc bits*(a: seq[byte], start, len: int): TrieBitSeq =
|
||||
doAssert start <= len
|
||||
doAssert len <= 8 * a.len
|
||||
TrieBitSeq(data: a, start: start, mLen: len)
|
||||
|
||||
template bits*(a: seq[byte]): TrieBitSeq =
|
||||
bits(a, 0, a.len * 8)
|
||||
|
||||
template bits*(a: seq[byte], len: int): TrieBitSeq =
|
||||
bits(a, 0, len)
|
||||
|
||||
template bits*(a: openArray[byte], start, len: int): TrieBitSeq =
|
||||
bits(@a, start, len)
|
||||
|
||||
template bits*(a: openArray[byte]): TrieBitSeq =
|
||||
bits(@a, 0, a.len * 8)
|
||||
|
||||
template bits*(a: openArray[byte], len: int): TrieBitSeq =
|
||||
bits(@a, 0, len)
|
||||
|
||||
template bits*(x: TrieBitSeq): TrieBitSeq = x
|
||||
|
||||
proc len*(r: TrieBitSeq): int = r.mLen
|
||||
|
||||
iterator enumerateBits(x: TrieBitSeq): (int, bool) =
|
||||
var p = x.start
|
||||
var i = 0
|
||||
let e = x.len
|
||||
while i != e:
|
||||
yield (i, getBitBE(x.data, p))
|
||||
inc p
|
||||
inc i
|
||||
|
||||
iterator items*(x: TrieBitSeq): bool =
|
||||
for _, v in enumerateBits(x): yield v
|
||||
|
||||
iterator pairs*(x: TrieBitSeq): (int, bool) =
|
||||
for i, v in enumerateBits(x): yield (i, v)
|
||||
|
||||
proc `[]`*(x: TrieBitSeq, idx: int): bool =
|
||||
doAssert idx < x.len
|
||||
let p = x.start + idx
|
||||
result = getBitBE(x.data, p)
|
||||
|
||||
proc sliceNormalized(x: TrieBitSeq, ibegin, iend: int): TrieBitSeq =
|
||||
doAssert ibegin >= 0 and
|
||||
ibegin < x.len and
|
||||
iend < x.len and
|
||||
iend + 1 >= ibegin # the +1 here allows the result to be
|
||||
# an empty range
|
||||
|
||||
result.data = x.data
|
||||
result.start = x.start + ibegin
|
||||
result.mLen = iend - ibegin + 1
|
||||
|
||||
proc `[]`*(r: TrieBitSeq, s: HSlice): TrieBitSeq =
|
||||
sliceNormalized(r, r @ s.a, r @ s.b)
|
||||
|
||||
proc `==`*(a, b: TrieBitSeq): bool =
|
||||
if a.len != b.len: return false
|
||||
for i in 0 ..< a.len:
|
||||
if a[i] != b[i]: return false
|
||||
true
|
||||
|
||||
proc `[]=`*(r: var TrieBitSeq, idx: Natural, val: bool) =
|
||||
doAssert idx < r.len
|
||||
let absIdx = r.start + idx
|
||||
changeBitBE(r.data, absIdx, val)
|
||||
|
||||
proc pushFront*(x: var TrieBitSeq, val: bool) =
|
||||
doAssert x.start > 0
|
||||
dec x.start
|
||||
x[0] = val
|
||||
inc x.mLen
|
||||
|
||||
template neededBytes(nBits: int): int =
|
||||
(nBits shr 3) + ord((nBits and 0b111) != 0)
|
||||
|
||||
static:
|
||||
doAssert neededBytes(2) == 1
|
||||
doAssert neededBytes(8) == 1
|
||||
doAssert neededBytes(9) == 2
|
||||
|
||||
proc `&`*(a, b: TrieBitSeq): TrieBitSeq =
|
||||
let totalLen = a.len + b.len
|
||||
|
||||
var bytes = newSeq[byte](totalLen.neededBytes)
|
||||
result = bits(bytes, 0, totalLen)
|
||||
|
||||
for i in 0 ..< a.len: result.data.changeBitBE(i, a[i])
|
||||
for i in 0 ..< b.len: result.data.changeBitBE(i + a.len, b[i])
|
||||
|
||||
proc `$`*(r: TrieBitSeq): string =
|
||||
result = newStringOfCap(r.len)
|
||||
for bit in r:
|
||||
result.add(if bit: '1' else: '0')
|
||||
|
||||
proc fromBits*(T: type, r: TrieBitSeq, offset, num: Natural): T =
|
||||
doAssert(num <= sizeof(T) * 8)
|
||||
# XXX: Nim has a bug that a typedesc parameter cannot be used
|
||||
# in a type coercion, so we must define an alias here:
|
||||
type TT = T
|
||||
for i in 0 ..< num:
|
||||
result = (result shl 1) or TT(r[offset + i])
|
||||
|
||||
proc parse*(T: type TrieBitSeq, s: string): TrieBitSeq =
|
||||
var bytes = newSeq[byte](s.len.neededBytes)
|
||||
for i, c in s:
|
||||
case c
|
||||
of '0': discard
|
||||
of '1': setBitBE(bytes, i)
|
||||
else: doAssert false
|
||||
result = bits(bytes, 0, s.len)
|
||||
|
||||
proc toBytes*(r: TrieBitSeq): seq[byte] =
|
||||
r.data[(r.start div 8)..<((r.mLen - r.start + 7) div 8)]
|
||||
@ -1,10 +0,0 @@
|
||||
import
|
||||
./trie_defs
|
||||
|
||||
export trie_defs
|
||||
|
||||
template checkValidHashZ*(x: untyped) =
|
||||
doAssert(x.len == 32 or x.len == 0)
|
||||
|
||||
template isZeroHash*(x: openArray[byte]): bool =
|
||||
x.len == 0
|
||||
@ -1,10 +1,5 @@
|
||||
import
|
||||
./test_bin_trie,
|
||||
./test_binaries_utils,
|
||||
./test_branches_utils,
|
||||
./test_examples,
|
||||
./test_hexary_trie,
|
||||
./test_json_suite,
|
||||
./test_transaction_db,
|
||||
./test_trie_bitseq,
|
||||
./test_hexary_proof
|
||||
|
||||
@ -1,128 +0,0 @@
|
||||
{.used.}
|
||||
|
||||
import
|
||||
std/random,
|
||||
unittest2,
|
||||
stew/byteutils,
|
||||
../../eth/trie/[db, binary],
|
||||
./testutils
|
||||
|
||||
suite "binary trie":
|
||||
|
||||
test "different order insert":
|
||||
randomize()
|
||||
var kv_pairs = randKVPair()
|
||||
var res = zeroHash
|
||||
for _ in 0..<1: # repeat 3 times
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
random.shuffle(kv_pairs)
|
||||
|
||||
for i, c in kv_pairs:
|
||||
trie.set(c.key, c.value)
|
||||
let x = trie.get(c.key)
|
||||
let y = c.value
|
||||
check y == x
|
||||
|
||||
check res == zeroHash or trie.getRootHash() == res
|
||||
res = trie.getRootHash()
|
||||
|
||||
# insert already exist key/value
|
||||
trie.set(kv_pairs[0].key, kv_pairs[0].value)
|
||||
check trie.getRootHash() == res
|
||||
|
||||
# Delete all key/value
|
||||
random.shuffle(kv_pairs)
|
||||
for i, c in kv_pairs:
|
||||
trie.delete(c.key)
|
||||
check trie.getRootHash() == zeroHash
|
||||
|
||||
const delSubtrieData = [
|
||||
(("\x12\x34\x56\x78", "78"), ("\x12\x34\x56\x79", "79"), "\x12\x34\x56", true, false),
|
||||
(("\x12\x34\x56\x78", "78"), ("\x12\x34\x56\xff", "ff"), "\x12\x34\x56", true, false),
|
||||
(("\x12\x34\x56\x78", "78"), ("\x12\x34\x56\x79", "79"), "\x12\x34\x57", false, false),
|
||||
(("\x12\x34\x56\x78", "78"), ("\x12\x34\x56\x79", "79"), "\x12\x34\x56\x78\x9a", false, true)
|
||||
]
|
||||
|
||||
test "delete subtrie":
|
||||
for data in delSubtrieData:
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
|
||||
let kv1 = data[0]
|
||||
let kv2 = data[1]
|
||||
let key_to_be_deleted = data[2]
|
||||
let will_delete = data[3]
|
||||
let will_raise_error = data[4]
|
||||
|
||||
# First test case, delete subtrie of a kv node
|
||||
trie.set(kv1[0].toBytes, kv1[1].toBytes)
|
||||
trie.set(kv2[0].toBytes, kv2[1].toBytes)
|
||||
check trie.get(kv1[0].toBytes) == kv1[1].toBytes
|
||||
check trie.get(kv2[0].toBytes) == kv2[1].toBytes
|
||||
|
||||
if will_delete:
|
||||
trie.deleteSubtrie(key_to_be_deleted.toBytes)
|
||||
check trie.get(kv1[0].toBytes) == []
|
||||
check trie.get(kv2[0].toBytes) == []
|
||||
check trie.getRootHash() == zeroHash
|
||||
else:
|
||||
if will_raise_error:
|
||||
try:
|
||||
trie.deleteSubtrie(key_to_be_deleted.toBytes)
|
||||
except NodeOverrideError:
|
||||
discard
|
||||
else:
|
||||
let root_hash_before_delete = trie.getRootHash()
|
||||
trie.deleteSubtrie(key_to_be_deleted.toBytes)
|
||||
check trie.get(kv1[0].toBytes) == toBytes(kv1[1])
|
||||
check trie.get(kv2[0].toBytes) == toBytes(kv2[1])
|
||||
check trie.getRootHash() == root_hash_before_delete
|
||||
|
||||
const invalidKeyData = [
|
||||
("\x12\x34\x56", false),
|
||||
("\x12\x34\x56\x77", false),
|
||||
("\x12\x34\x56\x78\x9a", true),
|
||||
("\x12\x34\x56\x79\xab", true),
|
||||
("\xab\xcd\xef", false)
|
||||
]
|
||||
|
||||
test "invalid key":
|
||||
for data in invalidKeyData:
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
|
||||
trie.set("\x12\x34\x56\x78".toBytes, "78".toBytes)
|
||||
trie.set("\x12\x34\x56\x79".toBytes, "79".toBytes)
|
||||
|
||||
let invalidKey = data[0]
|
||||
let if_error = data[1]
|
||||
|
||||
check trie.get(invalidKey.toBytes) == []
|
||||
|
||||
if if_error:
|
||||
try:
|
||||
trie.delete(invalidKey.toBytes)
|
||||
except NodeOverrideError:
|
||||
discard
|
||||
else:
|
||||
let previous_root_hash = trie.getRootHash()
|
||||
trie.delete(invalidKey.toBytes)
|
||||
check previous_root_hash == trie.getRootHash()
|
||||
|
||||
test "update value":
|
||||
let keys = randList(string, randGen(32, 32), randGen(100, 100))
|
||||
let vals = randList(int, randGen(0, 99), randGen(50, 50))
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
for key in keys:
|
||||
trie.set(key.toBytes, "old".toBytes)
|
||||
|
||||
var current_root = trie.getRootHash()
|
||||
for i in vals:
|
||||
trie.set(keys[i].toBytes, "old".toBytes)
|
||||
check current_root == trie.getRootHash()
|
||||
trie.set(keys[i].toBytes, "new".toBytes)
|
||||
check current_root != trie.getRootHash()
|
||||
check trie.get(keys[i].toBytes) == toBytes("new")
|
||||
current_root = trie.getRootHash()
|
||||
@ -1,179 +0,0 @@
|
||||
{.used.}
|
||||
|
||||
import
|
||||
std/strutils,
|
||||
unittest2,
|
||||
nimcrypto/[keccak, hash], stew/byteutils,
|
||||
../../eth/trie/[binaries, trie_bitseq],
|
||||
./testutils
|
||||
|
||||
proc parseBitVector(x: string): TrieBitSeq =
|
||||
result = genBitVec(x.len)
|
||||
for i, c in x:
|
||||
result[i] = (c == '1')
|
||||
|
||||
const
|
||||
commonPrefixData = [
|
||||
(@[0b0000_0000.byte], @[0b0000_0000.byte], 8),
|
||||
(@[0b0000_0000.byte], @[0b1000_0000.byte], 0),
|
||||
(@[0b1000_0000.byte], @[0b1100_0000.byte], 1),
|
||||
(@[0b0000_0000.byte], @[0b0100_0000.byte], 1),
|
||||
(@[0b1110_0000.byte], @[0b1100_0000.byte], 2),
|
||||
(@[0b0000_1111.byte], @[0b1111_1111.byte], 0)
|
||||
]
|
||||
|
||||
suite "binaries utils":
|
||||
|
||||
test "get common prefix length":
|
||||
for c in commonPrefixData:
|
||||
var
|
||||
c0 = c[0]
|
||||
c1 = c[1]
|
||||
let actual_a = getCommonPrefixLength(c0.bits, c1.bits)
|
||||
let actual_b = getCommonPrefixLength(c1.bits, c0.bits)
|
||||
let expected = c[2]
|
||||
check actual_a == actual_b
|
||||
check actual_a == expected
|
||||
|
||||
const
|
||||
None = ""
|
||||
parseNodeData = {
|
||||
"\x00\x03\x04\x05\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p":
|
||||
(0, "00110000010000000101", "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", false),
|
||||
"\x01\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p":
|
||||
(1, "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", false),
|
||||
"\x02value": (2, None, "value", false),
|
||||
"": (0, None, None, true),
|
||||
"\x00\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p": (0, None, None, true),
|
||||
"\x01\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p": (0, None, None, true),
|
||||
"\x01\x02\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p":
|
||||
(0, None, None, true),
|
||||
"\x02": (0, None, None, true),
|
||||
"\x03": (0, None, None, true)
|
||||
}
|
||||
|
||||
test "node parsing":
|
||||
for c in parseNodeData:
|
||||
let input = toBytes(c[0])
|
||||
let node = c[1]
|
||||
let kind = TrieNodeKind(node[0])
|
||||
let raiseError = node[3]
|
||||
var res: TrieNode
|
||||
|
||||
if raiseError:
|
||||
expect(InvalidNode):
|
||||
res = parseNode(input)
|
||||
else:
|
||||
res = parseNode(input)
|
||||
|
||||
check(kind == res.kind)
|
||||
case res.kind
|
||||
of KV_TYPE:
|
||||
check(res.keyPath == parseBitVector(node[1]))
|
||||
check(res.child == toBytes(node[2]))
|
||||
of BRANCH_TYPE:
|
||||
check(res.leftChild == toBytes(node[2]))
|
||||
check(res.rightChild == toBytes(node[2]))
|
||||
of LEAF_TYPE:
|
||||
check(res.value == toBytes(node[2]))
|
||||
|
||||
const
|
||||
kvData = [
|
||||
("0", "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", "\x00\x10\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", false),
|
||||
("" , "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", None, true),
|
||||
("0", "\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", None, true),
|
||||
("1", "\x00\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", None, true),
|
||||
("2", "", None, true)
|
||||
]
|
||||
|
||||
test "kv node encoding":
|
||||
for c in kvData:
|
||||
let keyPath = parseBitVector(c[0])
|
||||
let node = toBytes(c[1])
|
||||
let output = toBytes(c[2])
|
||||
let raiseError = c[3]
|
||||
|
||||
if raiseError:
|
||||
expect(ValidationError):
|
||||
check output == encodeKVNode(keyPath, node)
|
||||
else:
|
||||
check output == encodeKVNode(keyPath, node)
|
||||
|
||||
const
|
||||
branchData = [
|
||||
("\xc8\x9e\xfd\xaaT\xc0\xf2\x0cz\xdfa(\x82\xdf\tP\xf5\xa9Qc~\x03\x07\xcd\xcbLg/)\x8b\x8b\xc6", "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p",
|
||||
"\x01\xc8\x9e\xfd\xaaT\xc0\xf2\x0cz\xdfa(\x82\xdf\tP\xf5\xa9Qc~\x03\x07\xcd\xcbLg/)\x8b\x8b\xc6\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", false),
|
||||
("", "\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", None, true),
|
||||
("\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", "\x01", None, true),
|
||||
("\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", "12345", None, true),
|
||||
(repeat('\x01', 33), repeat('\x01', 32), None, true),
|
||||
]
|
||||
|
||||
test "branch node encode":
|
||||
for c in branchData:
|
||||
let left = toBytes(c[0])
|
||||
let right = toBytes(c[1])
|
||||
let output = toBytes(c[2])
|
||||
let raiseError = c[3]
|
||||
|
||||
if raiseError:
|
||||
expect(ValidationError):
|
||||
check output == encodeBranchNode(left, right)
|
||||
else:
|
||||
check output == encodeBranchNode(left, right)
|
||||
|
||||
const
|
||||
leafData = [
|
||||
("\x03\x04\x05", "\x02\x03\x04\x05", false),
|
||||
("", None, true)
|
||||
]
|
||||
|
||||
test "leaf node encode":
|
||||
for c in leafData:
|
||||
let raiseError = c[2]
|
||||
if raiseError:
|
||||
expect(ValidationError):
|
||||
check toBytes(c[1]) == encodeLeafNode(toBytes(c[0]))
|
||||
else:
|
||||
check toBytes(c[1]) == encodeLeafNode(toBytes(c[0]))
|
||||
|
||||
test "random kv encoding":
|
||||
let lengths = randList(int, randGen(1, 999), randGen(100, 100), unique = false)
|
||||
for len in lengths:
|
||||
var k = len
|
||||
var bitvec = genBitVec(len)
|
||||
var nodeHash = keccak256.digest(cast[ptr byte](k.addr), uint(sizeof(int)))
|
||||
var kvnode = encodeKVNode(bitvec, @(nodeHash.data))
|
||||
# first byte if KV_TYPE
|
||||
# in the middle are 1..n bits of binary-encoded-keypath
|
||||
# last 32 bytes are hash
|
||||
var keyPath = decodeToBinKeypath(kvnode[1..^33])
|
||||
check kvnode[0].ord == KV_TYPE.ord
|
||||
check keyPath == bitvec
|
||||
check kvnode[^32..^1] == nodeHash.data
|
||||
|
||||
test "optimized single bit keypath kvnode encoding":
|
||||
var k = 1
|
||||
var nodeHash = keccak256.digest(cast[ptr byte](k.addr), uint(sizeof(int)))
|
||||
var bitvec = genBitVec(1)
|
||||
bitvec[0] = false
|
||||
var kvnode = encodeKVNode(bitvec, @(nodeHash.data))
|
||||
var kp = decodeToBinKeypath(kvnode[1..^33])
|
||||
|
||||
var okv = encodeKVNode(false, @(nodeHash.data))
|
||||
check okv == kvnode
|
||||
var okp = decodeToBinKeypath(kvnode[1..^33])
|
||||
check okp == kp
|
||||
check okp.len == 1
|
||||
check okp == bitvec
|
||||
|
||||
bitvec[0] = true
|
||||
kvnode = encodeKVNode(bitvec, @(nodeHash.data))
|
||||
kp = decodeToBinKeypath(kvnode[1..^33])
|
||||
|
||||
okv = encodeKVNode(true, @(nodeHash.data))
|
||||
check okv == kvnode
|
||||
okp = decodeToBinKeypath(kvnode[1..^33])
|
||||
check okp == kp
|
||||
check okp.len == 1
|
||||
check okp == bitvec
|
||||
@ -1,146 +0,0 @@
|
||||
{.used.}
|
||||
|
||||
import
|
||||
std/[sets, strutils],
|
||||
unittest2,
|
||||
stew/byteutils,
|
||||
../../eth/trie/[db, binary, branches]
|
||||
|
||||
suite "branches utils":
|
||||
|
||||
proc testTrie(): BinaryTrie =
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
|
||||
trie.set("\x12\x34\x56\x78\x9a".toBytes, "9a".toBytes)
|
||||
trie.set("\x12\x34\x56\x78\x9b".toBytes, "9b".toBytes)
|
||||
trie.set("\x12\x34\x56\xff".toBytes, "ff".toBytes)
|
||||
trie
|
||||
|
||||
const branchExistData = [
|
||||
("\x12\x34", true),
|
||||
("\x12\x34\x56\x78\x9b", true),
|
||||
("\x12\x56", false),
|
||||
("\x12\x34\x56\xff\xff", false),
|
||||
("\x12\x34\x56", true),
|
||||
("\x12\x34\x56\x78", true)
|
||||
]
|
||||
|
||||
test "branch exists":
|
||||
var trie = testTrie()
|
||||
var db = trie.getDB()
|
||||
for c in branchExistData:
|
||||
let keyPrefix = c[0].toBytes
|
||||
let if_exist = c[1]
|
||||
check checkIfBranchExist(db, trie.getRootHash(), keyPrefix) == if_exist
|
||||
|
||||
const branchData = [
|
||||
("\x12\x34", true),
|
||||
("\x12\x34\x56\xff", true),
|
||||
("\x12\x34\x56\x78\x9b", true),
|
||||
("\x12\x56", true),
|
||||
("\x12\x34\x56\xff\xff", false),
|
||||
("", false)
|
||||
]
|
||||
|
||||
test "branch":
|
||||
var trie = testTrie()
|
||||
var db = trie.getDB()
|
||||
for c in branchData:
|
||||
let key = c[0].toBytes
|
||||
let keyValid = c[1]
|
||||
|
||||
if keyValid:
|
||||
let branch = getBranch(db, trie.getRootHash(), key)
|
||||
check isValidBranch(branch, trie.getRootHash(), key, trie.get(key))
|
||||
else:
|
||||
try:
|
||||
discard getBranch(db, trie.getRootHash(), key)
|
||||
except InvalidKeyError:
|
||||
check(true)
|
||||
except CatchableError:
|
||||
check(false)
|
||||
|
||||
const trieNodesData = [
|
||||
("#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xca", @["\x029a"]),
|
||||
("\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
@["\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x029a",
|
||||
"\x029b"]),
|
||||
("\x13\x07<\xa0w6\xd5O\x91\x93\xb1\xde,0}\xe7\xee\x82\xd7\xf6\xce\x1b^\xb7}\"\n\xe4&\xe2\xd7v",
|
||||
@["\x00\x82<M\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
"\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x029a",
|
||||
"\x029b"]),
|
||||
("X\x99\x8f\x13\xeb\x9bF\x08\xec|\x8b\xd8}\xca\xed\xda\xbb4\tl\xc8\x9bJ;J\xed\x11\x86\xc2\xd7+\xca",
|
||||
@["\x00\x80\x124V\xde\xb5\x8f\xdb\x98\xc0\xe8\xed\x10\xde\x84\x89\xe1\xc3\x90\xbeoi7y$sJ\x07\xa1h\xf5t\x1c\xac\r+",
|
||||
"\x01\x13\x07<\xa0w6\xd5O\x91\x93\xb1\xde,0}\xe7\xee\x82\xd7\xf6\xce\x1b^\xb7}\"\n\xe4&\xe2\xd7v7\x94\x07\x18\xc9\x96E\xf1\x9bS1sv\xa2\x8b\x9a\x88\xfd/>5\xcb3\x9e\x03\x08\r\xe2\xe1\xd5\xaaq",
|
||||
"\x00\x82<M\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
"\x00\x83\x7fR\xce\xe1\xe1 +\x96\xde\xae\xcdV\x13\x9a \x90.7H\xb6\x80\t\x10\xe1(\x03\x15\xde\x94\x17X\xee\xe1",
|
||||
"\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x02ff",
|
||||
"\x029a",
|
||||
"\x029b"]),
|
||||
("\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p", @[]),
|
||||
(repeat('0', 32), @[])
|
||||
]
|
||||
|
||||
proc toRanges(x: seq[string]): seq[seq[byte]] =
|
||||
result = newSeq[seq[byte]](x.len)
|
||||
for i, c in x: result[i] = toBytes(c)
|
||||
|
||||
test "get trie nodes":
|
||||
var trie = testTrie()
|
||||
var db = trie.getDB()
|
||||
for c in trieNodesData:
|
||||
let root = c[0].toBytes()
|
||||
let nodes = toRanges(c[1])
|
||||
check toHashSet(nodes) == toHashSet(getTrieNodes(db, root))
|
||||
|
||||
const witnessData = [
|
||||
("\x12\x34\x56\x78\x9b",
|
||||
@["\x00\x80\x124V\xde\xb5\x8f\xdb\x98\xc0\xe8\xed\x10\xde\x84\x89\xe1\xc3\x90\xbeoi7y$sJ\x07\xa1h\xf5t\x1c\xac\r+",
|
||||
"\x01\x13\x07<\xa0w6\xd5O\x91\x93\xb1\xde,0}\xe7\xee\x82\xd7\xf6\xce\x1b^\xb7}\"\n\xe4&\xe2\xd7v7\x94\x07\x18\xc9\x96E\xf1\x9bS1sv\xa2\x8b\x9a\x88\xfd/>5\xcb3\x9e\x03\x08\r\xe2\xe1\xd5\xaaq",
|
||||
"\x00\x82<M\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
"\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x029b"]),
|
||||
("\x12\x34\x56\x78",
|
||||
@["\x00\x80\x124V\xde\xb5\x8f\xdb\x98\xc0\xe8\xed\x10\xde\x84\x89\xe1\xc3\x90\xbeoi7y$sJ\x07\xa1h\xf5t\x1c\xac\r+",
|
||||
"\x01\x13\x07<\xa0w6\xd5O\x91\x93\xb1\xde,0}\xe7\xee\x82\xd7\xf6\xce\x1b^\xb7}\"\n\xe4&\xe2\xd7v7\x94\x07\x18\xc9\x96E\xf1\x9bS1sv\xa2\x8b\x9a\x88\xfd/>5\xcb3\x9e\x03\x08\r\xe2\xe1\xd5\xaaq",
|
||||
"\x00\x82<M\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
"\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x029a",
|
||||
"\x029b"]),
|
||||
("\x12\x34\x56",
|
||||
@["\x00\x80\x124V\xde\xb5\x8f\xdb\x98\xc0\xe8\xed\x10\xde\x84\x89\xe1\xc3\x90\xbeoi7y$sJ\x07\xa1h\xf5t\x1c\xac\r+",
|
||||
"\x01\x13\x07<\xa0w6\xd5O\x91\x93\xb1\xde,0}\xe7\xee\x82\xd7\xf6\xce\x1b^\xb7}\"\n\xe4&\xe2\xd7v7\x94\x07\x18\xc9\x96E\xf1\x9bS1sv\xa2\x8b\x9a\x88\xfd/>5\xcb3\x9e\x03\x08\r\xe2\xe1\xd5\xaaq",
|
||||
"\x00\x82<M\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
"\x00\x83\x7fR\xce\xe1\xe1 +\x96\xde\xae\xcdV\x13\x9a \x90.7H\xb6\x80\t\x10\xe1(\x03\x15\xde\x94\x17X\xee\xe1",
|
||||
"\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x02ff",
|
||||
"\x029a",
|
||||
"\x029b"]),
|
||||
("\x12",
|
||||
@["\x00\x80\x124V\xde\xb5\x8f\xdb\x98\xc0\xe8\xed\x10\xde\x84\x89\xe1\xc3\x90\xbeoi7y$sJ\x07\xa1h\xf5t\x1c\xac\r+",
|
||||
"\x01\x13\x07<\xa0w6\xd5O\x91\x93\xb1\xde,0}\xe7\xee\x82\xd7\xf6\xce\x1b^\xb7}\"\n\xe4&\xe2\xd7v7\x94\x07\x18\xc9\x96E\xf1\x9bS1sv\xa2\x8b\x9a\x88\xfd/>5\xcb3\x9e\x03\x08\r\xe2\xe1\xd5\xaaq",
|
||||
"\x00\x82<M\x84\x97\xc1\xf7S\xf5\xa2\xbb>\xbd\xe9\xc3t\x0f\xac/\xad\xa8\x01\xff\x9aE\t\xc1\xab\x9e\xa3|\xc7Z\xb0v",
|
||||
"\x00\x83\x7fR\xce\xe1\xe1 +\x96\xde\xae\xcdV\x13\x9a \x90.7H\xb6\x80\t\x10\xe1(\x03\x15\xde\x94\x17X\xee\xe1",
|
||||
"\x01#\xf037,w\xb9()\x0e4\x92\xdf\x11\xca\xea\xa5\x13/\x10\x1bJ\xa7\x16\x07\x07G\xb1\x01_\x16\xcaG\xe9\xb6\xa1\xfa\xd5\x82\xf4k\x04\x9c\x8e\xc8\x17\xb4G\xe1c*n\xf4o\x02\x85\xf1\x19\xa8\x83`\xfb\xf8\xa2",
|
||||
"\x02ff",
|
||||
"\x029a",
|
||||
"\x029b"]),
|
||||
(repeat('0', 32),
|
||||
@["\x00\x80\x124V\xde\xb5\x8f\xdb\x98\xc0\xe8\xed\x10\xde\x84\x89\xe1\xc3\x90\xbeoi7y$sJ\x07\xa1h\xf5t\x1c\xac\r+"]),
|
||||
]
|
||||
|
||||
test "get witness for key prefix":
|
||||
var trie = testTrie()
|
||||
var db = trie.getDB()
|
||||
for c in witnessData:
|
||||
let key = c[0].toBytes
|
||||
let nodes = toRanges(c[1])
|
||||
|
||||
if nodes.len != 0:
|
||||
let x = toHashSet(nodes)
|
||||
let y = toHashSet(getWitness(db, trie.getRootHash(), key))
|
||||
check x == y
|
||||
@ -1,100 +0,0 @@
|
||||
# Copyright (c) 2019-2022 Status Research & Development GmbH
|
||||
# Licensed and distributed under either of
|
||||
# * MIT license (license terms in the root directory or at http://opensource.org/licenses/MIT).
|
||||
# * Apache v2 license (license terms in the root directory or at http://www.apache.org/licenses/LICENSE-2.0).
|
||||
# at your option. This file may not be copied, modified, or distributed except according to those terms.
|
||||
|
||||
{.used.}
|
||||
|
||||
import
|
||||
unittest2,
|
||||
stew/byteutils,
|
||||
../../eth/trie/[db, binary, binaries, branches]
|
||||
|
||||
suite "examples":
|
||||
|
||||
var db = newMemoryDB()
|
||||
var trie = initBinaryTrie(db)
|
||||
|
||||
test "basic set/get":
|
||||
trie.set("key1".toBytes(), "value1".toBytes())
|
||||
trie.set("key2".toBytes(), "value2".toBytes())
|
||||
check trie.get("key1".toBytes) == "value1".toBytes
|
||||
check trie.get("key2".toBytes) == "value2".toBytes
|
||||
|
||||
test "check branch exists":
|
||||
check checkIfBranchExist(db, trie.getRootHash(), "key".toBytes) == true
|
||||
check checkIfBranchExist(db, trie.getRootHash(), "key1".toBytes) == true
|
||||
check checkIfBranchExist(db, trie.getRootHash(), "ken".toBytes) == false
|
||||
check checkIfBranchExist(db, trie.getRootHash(), "key123".toBytes) == false
|
||||
|
||||
test "branches utils":
|
||||
var branchA = getBranch(db, trie.getRootHash(), "key1".toBytes)
|
||||
# ==> [A, B, C1, D1]
|
||||
check branchA.len == 4
|
||||
|
||||
var branchB = getBranch(db, trie.getRootHash(), "key2".toBytes)
|
||||
# ==> [A, B, C2, D2]
|
||||
check branchB.len == 4
|
||||
|
||||
check isValidBranch(branchA, trie.getRootHash(), "key1".toBytes, "value1".toBytes) == true
|
||||
check isValidBranch(branchA, trie.getRootHash(), "key5".toBytes, "".toBytes) == true
|
||||
|
||||
expect InvalidNode:
|
||||
check isValidBranch(branchB, trie.getRootHash(), "key1".toBytes, "value1".toBytes)
|
||||
|
||||
var x = getBranch(db, trie.getRootHash(), "key".toBytes)
|
||||
# ==> [A]
|
||||
check x.len == 1
|
||||
|
||||
expect InvalidKeyError:
|
||||
x = getBranch(db, trie.getRootHash(), "key123".toBytes) # InvalidKeyError
|
||||
|
||||
x = getBranch(db, trie.getRootHash(), "key5".toBytes) # there is still branch for non-exist key
|
||||
# ==> [A]
|
||||
check x.len == 1
|
||||
|
||||
test "getWitness":
|
||||
var branch = getWitness(db, trie.getRootHash(), "key1".toBytes)
|
||||
# equivalent to `getBranch(db, trie.getRootHash(), "key1")`
|
||||
# ==> [A, B, C1, D1]
|
||||
check branch.len == 4
|
||||
|
||||
branch = getWitness(db, trie.getRootHash(), "key".toBytes)
|
||||
# this will include additional nodes of "key2"
|
||||
# ==> [A, B, C1, D1, C2, D2]
|
||||
check branch.len == 6
|
||||
|
||||
branch = getWitness(db, trie.getRootHash(), "".toBytes)
|
||||
# this will return the whole trie
|
||||
# ==> [A, B, C1, D1, C2, D2]
|
||||
check branch.len == 6
|
||||
|
||||
let beforeDeleteLen = db.totalRecordsInMemoryDB
|
||||
test "verify intermediate entries existence":
|
||||
var branchs = getWitness(db, trie.getRootHash, [])
|
||||
# set operation create new intermediate entries
|
||||
check branchs.len < beforeDeleteLen
|
||||
|
||||
var node = branchs[1]
|
||||
let nodeHash = keccak256(node)
|
||||
var nodes = getTrieNodes(db, @(nodeHash.data))
|
||||
check nodes.len == branchs.len - 1
|
||||
|
||||
test "delete sub trie":
|
||||
# delete all subtrie with key prefixes "key"
|
||||
trie.deleteSubtrie("key".toBytes)
|
||||
check trie.get("key1".toBytes) == []
|
||||
check trie.get("key2".toBytes) == []
|
||||
|
||||
test "prove the lie":
|
||||
# `delete` and `deleteSubtrie` not actually delete the nodes
|
||||
check db.totalRecordsInMemoryDB == beforeDeleteLen
|
||||
var branchs = getWitness(db, trie.getRootHash, [])
|
||||
check branchs.len == 0
|
||||
|
||||
test "dictionary syntax API":
|
||||
# dictionary syntax API
|
||||
trie["moon".toBytes] = "sun".toBytes
|
||||
check "moon".toBytes in trie
|
||||
check trie["moon".toBytes] == "sun".toBytes
|
||||
@ -1,85 +0,0 @@
|
||||
{.used.}
|
||||
|
||||
import
|
||||
std/random,
|
||||
unittest2,
|
||||
../../eth/trie/trie_bitseq
|
||||
|
||||
proc randomBytes(n: int): seq[byte] =
|
||||
result = newSeq[byte](n)
|
||||
for i in 0 ..< result.len:
|
||||
result[i] = byte(rand(256))
|
||||
|
||||
suite "trie bitseq":
|
||||
test "basic":
|
||||
var a = @[byte 0b10101010, 0b11110000, 0b00001111, 0b01010101]
|
||||
|
||||
var bSeq = @[byte 0b10101010, 0b00000000, 0b00000000, 0b11111111]
|
||||
var b = bits(bSeq, 8)
|
||||
|
||||
var cSeq = @[byte 0b11110000, 0b00001111, 0b00000000, 0b00000000]
|
||||
var c = bits(cSeq, 16)
|
||||
|
||||
var dSeq = @[byte 0b00001111, 0b00000000, 0b00000000, 0b00000000]
|
||||
var d = bits(dSeq, 8)
|
||||
|
||||
var eSeq = @[byte 0b01010101, 0b00000000, 0b00000000, 0b00000000]
|
||||
var e = bits(eSeq, 8)
|
||||
|
||||
var m = a.bits
|
||||
var n = m[0..7]
|
||||
check n == b
|
||||
check n.len == 8
|
||||
check b.len == 8
|
||||
check c == m[8..23]
|
||||
check $(d) == "00001111"
|
||||
check $(e) == "01010101"
|
||||
|
||||
var f = int.fromBits(e, 0, 4)
|
||||
check f == 0b0101
|
||||
|
||||
let k = n & d
|
||||
check(k.len == n.len + d.len)
|
||||
check($k == $n & $d)
|
||||
|
||||
var asciiSeq = @[byte('A'),byte('S'),byte('C'),byte('I'),byte('I')]
|
||||
let asciiBits = bits(asciiSeq)
|
||||
check $asciiBits == "0100000101010011010000110100100101001001"
|
||||
|
||||
test "concat operator":
|
||||
randomize(5000)
|
||||
|
||||
for i in 0..<256:
|
||||
var xSeq = randomBytes(rand(i))
|
||||
var ySeq = randomBytes(rand(i))
|
||||
let x = xSeq.bits
|
||||
let y = ySeq.bits
|
||||
var z = x & y
|
||||
check z.len == x.len + y.len
|
||||
check($z == $x & $y)
|
||||
|
||||
test "get set bits":
|
||||
randomize(1000)
|
||||
|
||||
for i in 0..<256:
|
||||
# produce random vector
|
||||
var xSeq = randomBytes(i)
|
||||
var ySeq = randomBytes(i)
|
||||
var x = xSeq.bits
|
||||
var y = ySeq.bits
|
||||
for idx, bit in x:
|
||||
y[idx] = bit
|
||||
check x == y
|
||||
|
||||
test "constructor with start":
|
||||
var a = @[byte 0b10101010, 0b11110000, 0b00001111, 0b01010101]
|
||||
var b = a.bits(1, 8)
|
||||
check b.len == 8
|
||||
check b[0] == false
|
||||
check $b == "01010101"
|
||||
b[0] = true
|
||||
check $b == "11010101"
|
||||
check b[0] == true
|
||||
b.pushFront(false)
|
||||
check b[0] == false
|
||||
check $b == "011010101"
|
||||
@ -1,7 +1,6 @@
|
||||
import
|
||||
std/[random, sets],
|
||||
nimcrypto/[utils, sysrand],
|
||||
../../eth/trie/trie_bitseq
|
||||
nimcrypto/[utils, sysrand]
|
||||
|
||||
type
|
||||
RandGen*[T] = object
|
||||
@ -77,10 +76,3 @@ proc randKVPair*(keySize = 32): seq[KVPair] =
|
||||
result = newSeq[KVPair](listLen)
|
||||
for i in 0..<listLen:
|
||||
result[i] = KVPair(key: keys[i], value: vals[i])
|
||||
|
||||
proc genBitVec*(len: int): TrieBitSeq =
|
||||
let k = ((len + 7) and (not 7)) shr 3
|
||||
var s = newSeq[byte](k)
|
||||
result = bits(s, len)
|
||||
for i in 0..<len:
|
||||
result[i] = rand(2) == 1
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user