status-im/nimbus-eth1
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth1.git synced 2025-02-23 01:08:26 +00:00
Code Issues Projects Releases Wiki Activity

Aristo uses pre classified tree types (#2385)

Browse Source 
* Remove unused `merge*()` functions (for production)

details:
  Some functionality moved to test suite

* Make sure that only `AccountData` leaf type is exactly used on VertexID(1)

* clean up payload type

* Provide dedicated functions for merging accounts and storage trees

why:
  Storage trees are always linked to an account, so there is no need
  for an application to fiddle about (e.e. creating, re-cycling) with
  storage tree vertex IDs.

* CoreDb: Disable tracer functionality

why:
  Must be updated to accommodate new/changed `Aristo` functions.

* CoreDb: Use new `mergeXXX()` functions

why:
  Makes explicit vertex ID management obsolete for creating new
  storage trees.

* Remove `mergePayload()` and other cruft from API, `aristo_merge`, etc.

* clean up merge functions

details:
  The merge implementation `mergePayloadImpl()` does not need to be super
  generic anymore as all the edge cases are covered by the specialised
  functions `mergeAccountPayload()`, `mergeGenericData()`, and
  `mergeStorageData()`.

* No tracer available at the moment, so disable offending tests
This commit is contained in:
Jordan Hrycaj 2024-06-18 11:14:02 +00:00 committed by GitHub
parent e3d14bd921
commit 51f02090b8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
24 changed files with 579 additions and 592 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
nimbus/TODO-TRACER.md Normal file
View File

@ -0,0 +1,7 @@
The `handlers_tracer` driver from the `CoreDb` module needs to be re-factored.
This module will slightly change its work modus and will run as a genuine
logger. The previously available restore features were ill concieved, an
attempt to be as close as possible to the legacy tracer. If resoring is
desired the tracer will need to run inside a transaction (which it does
anyway.)

42
nimbus/db/aristo/README.md
View File

@ -22,11 +22,10 @@ Contents
+ [4.2 Extension record serialisation](#ch4x2) + [4.2 Extension record serialisation](#ch4x2)
+ [4.3 Leaf record serialisation](#ch4x3) + [4.3 Leaf record serialisation](#ch4x3)
+ [4.4 Leaf record payload serialisation for account data](#ch4x4) + [4.4 Leaf record payload serialisation for account data](#ch4x4)
+ [4.5 Leaf record payload serialisation for RLP encoded data](#ch4x5) + [4.5 Leaf record payload serialisation for unstructured data](#ch4x5)
+ [4.6 Leaf record payload serialisation for unstructured data](#ch4x6) + [4.6 Serialisation of the top used vertex ID](#ch4x6)
+ [4.7 Serialisation of the top used vertex ID](#ch4x7) + [4.7 Serialisation of a last saved state record](#ch4x7)
+ [4.8 Serialisation of a last saved state record](#ch4x8) + [4.8 Serialisation record identifier identification](#ch4x8)
+ [4.9 Serialisation record identifier identification](#ch4x9)
* [5. *Patricia Trie* implementation notes](#ch5) * [5. *Patricia Trie* implementation notes](#ch5)
+ [5.1 Database decriptor representation](#ch5x1) + [5.1 Database decriptor representation](#ch5x1)
@ -327,19 +326,7 @@ fields. So, joining the *4 x bitmask(2)* word array to a single byte, the
maximum value of that byte is 0x99. maximum value of that byte is 0x99.
<a name="ch4x5"></a> <a name="ch4x5"></a>
### 4.5 Leaf record payload serialisation for RLP encoded data ### 4.5 Leaf record payload serialisation for unstructured data
0 +--+ .. --+
| | | -- data, at least one byte
+--+ .. --+
| | -- marker(8), 0x6a
+--+
where
marker(8) is the eight bit array *0110-1010*
<a name="ch4x6"></a>
### 4.6 Leaf record payload serialisation for unstructured data
0 +--+ .. --+ 0 +--+ .. --+
| | | -- data, at least one byte | | | -- data, at least one byte
@ -350,8 +337,8 @@ maximum value of that byte is 0x99.
where where
marker(8) is the eight bit array *0110-1011* marker(8) is the eight bit array *0110-1011*
<a name="ch4x7"></a> <a name="ch4x6"></a>
### 4.7 Serialisation of the top used vertex ID ### 4.6 Serialisation of the top used vertex ID
0 +--+--+--+--+--+--+--+--+ 0 +--+--+--+--+--+--+--+--+
| | -- last used vertex IDs | | -- last used vertex IDs
@ -367,8 +354,8 @@ indicates that all ID values greater or equal than this value are free and can
be used as vertex IDs. If this record is missing, the value *(1u64,0x01)* is be used as vertex IDs. If this record is missing, the value *(1u64,0x01)* is
assumed, i.e. the list with the single vertex ID *1*. assumed, i.e. the list with the single vertex ID *1*.
<a name="ch4x8"></a> <a name="ch4x7"></a>
### 4.8 Serialisation of a last saved state record ### 4.7 Serialisation of a last saved state record
0 +--+--+--+--+--+ .. --+--+ .. --+ 0 +--+--+--+--+--+ .. --+--+ .. --+
| | -- 32 bytes source state hash | | -- 32 bytes source state hash
@ -383,8 +370,8 @@ assumed, i.e. the list with the single vertex ID *1*.
where where
marker(8) is the eight bit array *0111-111f* marker(8) is the eight bit array *0111-111f*
<a name="ch4x10"></a> <a name="ch4x8"></a>
### 4.9 Serialisation record identifier tags ### 4.8 Serialisation record identifier tags
Any of the above records can uniquely be identified by its trailing marker, Any of the above records can uniquely be identified by its trailing marker,
i.e. the last byte of a serialised record. i.e. the last byte of a serialised record.
@ -395,10 +382,9 @@ i.e. the last byte of a serialised record.
| 10xx xxxx | 0x80 + x(6) | Extension record | [4.2](#ch4x2) | | 10xx xxxx | 0x80 + x(6) | Extension record | [4.2](#ch4x2) |
| 11xx xxxx | 0xC0 + x(6) | Leaf record | [4.3](#ch4x3) | | 11xx xxxx | 0xC0 + x(6) | Leaf record | [4.3](#ch4x3) |
| 0xxx 0yyy | (x(3)<<4) + y(3) | Account payload | [4.4](#ch4x4) | | 0xxx 0yyy | (x(3)<<4) + y(3) | Account payload | [4.4](#ch4x4) |
| 0110 1010 | 0x6a | RLP encoded payload | [4.5](#ch4x5) | | 0110 1011 | 0x6b | Unstructured payload | [4.5](#ch4x5) |
| 0110 1011 | 0x6b | Unstructured payload | [4.6](#ch4x6) | | 0111 1100 | 0x7c | Last used vertex ID | [4.6](#ch4x6) |
| 0111 1100 | 0x7c | Last used vertex ID | [4.7](#ch4x7) | | 0111 1111 | 0x7f | Last saved state | [4.7](#ch4x7) |
| 0111 1111 | 0x7f | Last saved state | [4.8](#ch4x8) |
<a name="ch5"></a> <a name="ch5"></a>
5. *Patricia Trie* implementation notes 5. *Patricia Trie* implementation notes

155
nimbus/db/aristo/aristo_api.nim
View File

@ -20,7 +20,7 @@ import
./aristo_init/memory_db, ./aristo_init/memory_db,
"."/[aristo_delete, aristo_desc, aristo_fetch, aristo_get, aristo_hashify, "."/[aristo_delete, aristo_desc, aristo_fetch, aristo_get, aristo_hashify,
aristo_hike, aristo_init, aristo_merge, aristo_path, aristo_profile, aristo_hike, aristo_init, aristo_merge, aristo_path, aristo_profile,
aristo_serialise, aristo_tx, aristo_vid] aristo_serialise, aristo_tx]
export export
AristoDbProfListRef AristoDbProfListRef
@ -100,7 +100,8 @@ type
): Result[PayloadRef,(VertexID,AristoError)] ): Result[PayloadRef,(VertexID,AristoError)]
{.noRaise.} {.noRaise.}
## Cascaded attempt to traverse the `Aristo Trie` and fetch the value ## Cascaded attempt to traverse the `Aristo Trie` and fetch the value
## of a leaf vertex. This function is complementary to `mergePayload()`. ## of a leaf vertex. This function is complementary to some `mergeXXX()`
## function.
AristoApiFindTxFn* = AristoApiFindTxFn* =
proc(db: AristoDbRef; proc(db: AristoDbRef;
@ -230,34 +231,45 @@ type
## `reCentre()` for details.) This function is a fast version of ## `reCentre()` for details.) This function is a fast version of
## `db.forked.toSeq.len`. ## `db.forked.toSeq.len`.
AristoApiMergeFn* = AristoApiMergeAccountPayloadFn* =
proc(db: AristoDbRef; proc(db: AristoDbRef;
root: VertexID; accPath: openArray[byte];
path: openArray[byte]; accPayload: AristoAccount;
data: openArray[byte];
accPath: PathID;
): Result[bool,AristoError] ): Result[bool,AristoError]
{.noRaise.} {.noRaise.}
## Veriant of `mergePayload()` where the `data` argument will be ## Merge the key-value-pair argument `(accKey,accPayload)` as an account
## converted to a `RawBlob` type `PayloadRef` value. ## ledger value, i.e. the the sub-tree starting at `VertexID(1)`.
AristoApiMergePayloadFn* =
proc(db: AristoDbRef;
root: VertexID;
path: openArray[byte];
payload: PayloadRef;
accPath = VOID_PATH_ID;
): Result[bool,AristoError]
{.noRaise.}
## Merge the argument key-value-pair `(path,payload)` into the top level
## vertex table of the database `db`.
## ##
## For a `root` argument with `VertexID` greater than `LEAST_FREE_VID`, ## The payload argument `accPayload` must have the `storageID` field
## the sub-tree generated by `payload.root` is considered a storage trie ## either unset/invalid or referring to a existing vertex which will be
## linked to an account leaf referred to by a valid `accPath` (i.e. ## assumed to be a storage tree.
## different from `VOID_PATH_ID`.) In that case, an account must exists.
## If there is payload of type `AccountData`, its `storageID` field must AristoApiMergeGenericDataFn* =
## be unset or equal to the `payload.root` vertex ID. proc( db: AristoDbRef;
root: VertexID;
path: openArray[byte];
data: openArray[byte];
): Result[bool,AristoError]
{.noRaise.}
## Variant of `mergeXXX()` for generic sub-trees, i.e. for arguments
## `root` greater than `VertexID(1)` and smaller than `LEAST_FREE_VID`.
AristoApiMergeStorageDataFn* =
proc(db: AristoDbRef;
stoKey: openArray[byte];
stoData: openArray[byte];
accPath: PathID;
): Result[VertexID,AristoError]
{.noRaise.}
## Merge the key-value-pair argument `(stoKey,stoData)` as a storage
## value. This means, the root vertex will be derived from the `accPath`
## argument, the Patricia tree path for the storage tree is given by
## `stoKey` and the leaf value with the payload will be stored as a
## `PayloadRef` object of type `RawData`.
##
## If the storage tree does not exist yet it will be created and the
## payload leaf accessed by `accPath` will be updated with the storage
## tree vertex ID.
AristoApiPathAsBlobFn* = AristoApiPathAsBlobFn* =
proc(tag: PathID; proc(tag: PathID;
@ -347,28 +359,6 @@ type
{.noRaise.} {.noRaise.}
## Getter, returns top level transaction if there is any. ## Getter, returns top level transaction if there is any.
AristoApiVidFetchFn* =
proc(db: AristoDbRef;
pristine = false;
): VertexID
{.noRaise.}
## Recycle or create a new `VertexID`. Reusable vertex *ID*s are kept
## in a list where the top entry *ID* has the property that any other
## *ID* larger is also not used on the database.
##
## The function prefers to return recycled vertex *ID*s if there are
## any. When the argument `pristine` is set `true`, the function
## guarantees to return a non-recycled, brand new vertex *ID* which
## is the preferred mode when creating leaf vertices.
AristoApiVidDisposeFn* =
proc(db: AristoDbRef;
vid: VertexID;
) {.noRaise.}
## Recycle the argument `vtxID` which is useful after deleting entries
## from the vertex table to prevent the `VertexID` type key values
## small.
AristoApiRef* = ref AristoApiObj AristoApiRef* = ref AristoApiObj
AristoApiObj* = object of RootObj AristoApiObj* = object of RootObj
## Useful set of `Aristo` fuctions that can be filtered, stacked etc. ## Useful set of `Aristo` fuctions that can be filtered, stacked etc.
@ -388,8 +378,9 @@ type
isTop*: AristoApiIsTopFn isTop*: AristoApiIsTopFn
level*: AristoApiLevelFn level*: AristoApiLevelFn
nForked*: AristoApiNForkedFn nForked*: AristoApiNForkedFn
merge*: AristoApiMergeFn mergeAccountPayload*: AristoApiMergeAccountPayloadFn
mergePayload*: AristoApiMergePayloadFn mergeGenericData*: AristoApiMergeGenericDataFn
mergeStorageData*: AristoApiMergeStorageDataFn
pathAsBlob*: AristoApiPathAsBlobFn pathAsBlob*: AristoApiPathAsBlobFn
persist*: AristoApiPersistFn persist*: AristoApiPersistFn
reCentre*: AristoApiReCentreFn reCentre*: AristoApiReCentreFn
@ -397,8 +388,6 @@ type
serialise*: AristoApiSerialiseFn serialise*: AristoApiSerialiseFn
txBegin*: AristoApiTxBeginFn txBegin*: AristoApiTxBeginFn
txTop*: AristoApiTxTopFn txTop*: AristoApiTxTopFn
vidFetch*: AristoApiVidFetchFn
vidDispose*: AristoApiVidDisposeFn
AristoApiProfNames* = enum AristoApiProfNames* = enum
@ -421,8 +410,9 @@ type
AristoApiProfIsTopFn = "isTop" AristoApiProfIsTopFn = "isTop"
AristoApiProfLevelFn = "level" AristoApiProfLevelFn = "level"
AristoApiProfNForkedFn = "nForked" AristoApiProfNForkedFn = "nForked"
AristoApiProfMergeFn = "merge" AristoApiProfMergeAccountPayloadFn = "mergeAccountPayload"
AristoApiProfMergePayloadFn = "mergePayload" AristoApiProfMergeGenericDataFn = "mergeGenericData"
AristoApiProfMergeStorageDataFn = "mergeStorageData"
AristoApiProfPathAsBlobFn = "pathAsBlob" AristoApiProfPathAsBlobFn = "pathAsBlob"
AristoApiProfPersistFn = "persist" AristoApiProfPersistFn = "persist"
AristoApiProfReCentreFn = "reCentre" AristoApiProfReCentreFn = "reCentre"
@ -430,8 +420,6 @@ type
AristoApiProfSerialiseFn = "serialise" AristoApiProfSerialiseFn = "serialise"
AristoApiProfTxBeginFn = "txBegin" AristoApiProfTxBeginFn = "txBegin"
AristoApiProfTxTopFn = "txTop" AristoApiProfTxTopFn = "txTop"
AristoApiProfVidFetchFn = "vidFetch"
AristoApiProfVidDisposeFn = "vidDispose"
AristoApiProfBeGetVtxFn = "be/getVtx" AristoApiProfBeGetVtxFn = "be/getVtx"
AristoApiProfBeGetKeyFn = "be/getKey" AristoApiProfBeGetKeyFn = "be/getKey"
@ -470,8 +458,9 @@ when AutoValidateApiHooks:
doAssert not api.isTop.isNil doAssert not api.isTop.isNil
doAssert not api.level.isNil doAssert not api.level.isNil
doAssert not api.nForked.isNil doAssert not api.nForked.isNil
doAssert not api.merge.isNil doAssert not api.mergeAccountPayload.isNil
doAssert not api.mergePayload.isNil doAssert not api.mergeGenericData.isNil
doAssert not api.mergeStorageData.isNil
doAssert not api.pathAsBlob.isNil doAssert not api.pathAsBlob.isNil
doAssert not api.persist.isNil doAssert not api.persist.isNil
doAssert not api.reCentre.isNil doAssert not api.reCentre.isNil
@ -479,8 +468,6 @@ when AutoValidateApiHooks:
doAssert not api.serialise.isNil doAssert not api.serialise.isNil
doAssert not api.txBegin.isNil doAssert not api.txBegin.isNil
doAssert not api.txTop.isNil doAssert not api.txTop.isNil
doAssert not api.vidFetch.isNil
doAssert not api.vidDispose.isNil
proc validate(prf: AristoApiProfRef) = proc validate(prf: AristoApiProfRef) =
prf.AristoApiRef.validate prf.AristoApiRef.validate
@ -523,8 +510,9 @@ func init*(api: var AristoApiObj) =
api.isTop = isTop api.isTop = isTop
api.level = level api.level = level
api.nForked = nForked api.nForked = nForked
api.merge = merge api.mergeAccountPayload = mergeAccountPayload
api.mergePayload = mergePayload api.mergeGenericData = mergeGenericData
api.mergeStorageData = mergeStorageData
api.pathAsBlob = pathAsBlob api.pathAsBlob = pathAsBlob
api.persist = persist api.persist = persist
api.reCentre = reCentre api.reCentre = reCentre
@ -532,8 +520,6 @@ func init*(api: var AristoApiObj) =
api.serialise = serialise api.serialise = serialise
api.txBegin = txBegin api.txBegin = txBegin
api.txTop = txTop api.txTop = txTop
api.vidFetch = vidFetch
api.vidDispose = vidDispose
when AutoValidateApiHooks: when AutoValidateApiHooks:
api.validate api.validate
@ -559,17 +545,16 @@ func dup*(api: AristoApiRef): AristoApiRef =
isTop: api.isTop, isTop: api.isTop,
level: api.level, level: api.level,
nForked: api.nForked, nForked: api.nForked,
merge: api.merge, mergeAccountPayload: api.mergeAccountPayload,
mergePayload: api.mergePayload, mergeGenericData: api.mergeGenericData,
mergeStorageData: api.mergeStorageData,
pathAsBlob: api.pathAsBlob, pathAsBlob: api.pathAsBlob,
persist: api.persist, persist: api.persist,
reCentre: api.reCentre, reCentre: api.reCentre,
rollback: api.rollback, rollback: api.rollback,
serialise: api.serialise, serialise: api.serialise,
txBegin: api.txBegin, txBegin: api.txBegin,
txTop: api.txTop, txTop: api.txTop)
vidFetch: api.vidFetch,
vidDispose: api.vidDispose)
when AutoValidateApiHooks: when AutoValidateApiHooks:
api.validate api.validate
@ -680,16 +665,20 @@ func init*(
AristoApiProfNForkedFn.profileRunner: AristoApiProfNForkedFn.profileRunner:
result = api.nForked(a) result = api.nForked(a)
profApi.merge = profApi.mergeAccountPayload =
proc(a: AristoDbRef; b: VertexID; c,d: openArray[byte]; e: PathID): auto = proc(a: AristoDbRef; b, c: openArray[byte]): auto =
AristoApiProfMergeFn.profileRunner: AristoApiProfMergeAccountPayloadFn.profileRunner:
result = api.merge(a, b, c, d ,e) result = api.mergeAccountPayload(a, b, c)
profApi.mergePayload = profApi.mergeGenericData =
proc(a: AristoDbRef; b: VertexID; c: openArray[byte]; d: PayloadRef; proc(a: AristoDbRef; b: VertexID, c, d: openArray[byte]): auto =
e = VOID_PATH_ID): auto = AristoApiProfMergeGenericDataFn.profileRunner:
AristoApiProfMergePayloadFn.profileRunner: result = api.mergeGenericData(a, b, c, d)
result = api.mergePayload(a, b, c, d ,e)
profApi.mergeStorageData =
proc(a: AristoDbRef; b, c: openArray[byte]; d: PathID): auto =
AristoApiProfMergeStorageDataFn.profileRunner:
result = api.mergeStorageData(a, b, c, d)
profApi.pathAsBlob = profApi.pathAsBlob =
proc(a: PathID): auto = proc(a: PathID): auto =
@ -726,16 +715,6 @@ func init*(
AristoApiProfTxTopFn.profileRunner: AristoApiProfTxTopFn.profileRunner:
result = api.txTop(a) result = api.txTop(a)
profApi.vidFetch =
proc(a: AristoDbRef; b = false): auto =
AristoApiProfVidFetchFn.profileRunner:
result = api.vidFetch(a, b)
profApi.vidDispose =
proc(a: AristoDbRef;b: VertexID) =
AristoApiProfVidDisposeFn.profileRunner:
api.vidDispose(a, b)
let beDup = be.dup() let beDup = be.dup()
if beDup.isNil: if beDup.isNil:
profApi.be = be profApi.be = be

6
nimbus/db/aristo/aristo_blobify.nim
View File

@ -46,9 +46,6 @@ proc blobifyTo*(pyl: PayloadRef, data: var Blob) =
of RawData: of RawData:
data &= pyl.rawBlob data &= pyl.rawBlob
data &= [0x6b.byte] data &= [0x6b.byte]
of RlpData:
data &= pyl.rlpBlob
data &= @[0x6a.byte]
of AccountData: of AccountData:
var mask: byte var mask: byte
@ -194,9 +191,6 @@ proc deblobifyTo(
if mask == 0x6b: # unstructured payload if mask == 0x6b: # unstructured payload
pyl = PayloadRef(pType: RawData, rawBlob: data[0 .. ^2]) pyl = PayloadRef(pType: RawData, rawBlob: data[0 .. ^2])
return ok() return ok()
if mask == 0x6a: # RLP encoded payload
pyl = PayloadRef(pType: RlpData, rlpBlob: data[0 .. ^2])
return ok()
var var
pAcc = PayloadRef(pType: AccountData) pAcc = PayloadRef(pType: AccountData)

2
nimbus/db/aristo/aristo_debug.nim
View File

@ -210,8 +210,6 @@ proc ppPayload(p: PayloadRef, db: AristoDbRef): string =
case p.pType: case p.pType:
of RawData: of RawData:
result &= p.rawBlob.toHex.squeeze(hex=true) result &= p.rawBlob.toHex.squeeze(hex=true)
of RlpData:
result &= "[#" & p.rlpBlob.toHex.squeeze(hex=true) & "]"
of AccountData: of AccountData:
result = "(" result = "("
result &= ($p.account.nonce).stripZeros(toExp=true) & "," result &= ($p.account.nonce).stripZeros(toExp=true) & ","

47
nimbus/db/aristo/aristo_desc/desc_error.nim
View File

@ -86,47 +86,38 @@ type
# Merge leaf `merge()` # Merge leaf `merge()`
MergeAssemblyFailed # Ooops, internal error MergeAssemblyFailed # Ooops, internal error
MergeAccRootNotAccepted
MergeStoRootNotAccepted
MergeBranchGarbledNibble MergeBranchGarbledNibble
MergeBranchGarbledTail MergeBranchGarbledTail
MergeBranchLinkLeafGarbled MergeBranchLinkLeafGarbled
MergeBranchLinkLockedKey
MergeBranchLinkProofModeLock
MergeBranchLinkVtxPfxTooShort MergeBranchLinkVtxPfxTooShort
MergeBranchProofModeLock MergeBranchProofModeLock
MergeBranchRootExpected MergeBranchRootExpected
MergeLeafCantChangePayloadType MergeHashKeyDiffersFromCached
MergeHashKeyInvalid
MergeLeafCantChangeStorageID MergeLeafCantChangeStorageID
MergeLeafGarbledHike MergeLeafGarbledHike
MergeLeafPathCachedAlready MergeLeafPathCachedAlready
MergeLeafPathOnBackendAlready MergeLeafPathOnBackendAlready
MergeLeafProofModeLock MergeLeafProofModeLock
MergeLeafTypeAccountRequired
MergeLeafTypeRawDataRequired
MergeNodeAccountPayloadError
MergeNodeVidMissing
MergeNodeVtxDiffersFromExisting
MergeNonBranchProofModeLock MergeNonBranchProofModeLock
MergeProofInitMissing
MergeRevVidMustHaveBeenCached
MergeRootArgsIncomplete
MergeRootBranchLinkBusy MergeRootBranchLinkBusy
MergeRootMissing MergeRootKeyDiffersForVid
MergeHashKeyInvalid
MergeHashKeyDiffersFromCached
MergeHashKeyRevLookUpGarbled
MergeRootKeyInvalid MergeRootKeyInvalid
MergeRootKeyMissing
MergeRootKeyNotInProof MergeRootKeyNotInProof
MergeRootKeysMissing MergeRootKeysMissing
MergeRootKeysOverflow MergeRootKeysOverflow
MergeProofInitMissing MergeRootVidMissing
MergeRevVidMustHaveBeenCached
MergeNodeVtxDiffersFromExisting
MergeNodeVidMissing
MergeNodeAccountPayloadError
MergeRootKeyDiffersForVid
MergeNodeVtxDuplicates
MergeRootKeyMissing
MergeRootArgsIncomplete
# Utils
UtilsAccPathMissing
UtilsAccPathWithoutLeaf
UtilsAccUnaccessible
UtilsAccWrongStorageRoot
UtilsStoRootMissing
# Update `Merkle` hashes `hashify()` # Update `Merkle` hashes `hashify()`
HashifyVtxUnresolved HashifyVtxUnresolved
@ -289,6 +280,14 @@ type
AccNodeUnsupported AccNodeUnsupported
PayloadTypeUnsupported PayloadTypeUnsupported
UtilsAccPathMissing
UtilsAccPathWithoutLeaf
UtilsAccInaccessible
UtilsAccWrongStorageRoot
UtilsStoRootInaccessible
UtilsStoRootMissing
UtilsAccLeafPayloadExpected
# Miscelaneous handy helpers # Miscelaneous handy helpers
AccRootUnacceptable AccRootUnacceptable
MptRootUnacceptable MptRootUnacceptable

13
nimbus/db/aristo/aristo_desc/desc_structural.nim
View File

@ -41,15 +41,15 @@ type
PayloadType* = enum PayloadType* = enum
## Type of leaf data. ## Type of leaf data.
RawData ## Generic data RawData ## Generic data
RlpData ## Marked RLP encoded
AccountData ## `Aristo account` with vertex IDs links AccountData ## `Aristo account` with vertex IDs links
PayloadRef* = ref object of RootRef PayloadRef* = ref object of RootRef
## The payload type depends on the sub-tree used. The `VertesID(1)` rooted
## sub-tree only has `AccountData` type payload, while all other sub-trees
## have `RawData` payload.
case pType*: PayloadType case pType*: PayloadType
of RawData: of RawData:
rawBlob*: Blob ## Opaque data, default value rawBlob*: Blob ## Opaque data, default value
of RlpData:
rlpBlob*: Blob ## Opaque data marked RLP encoded
of AccountData: of AccountData:
account*: AristoAccount account*: AristoAccount
@ -162,9 +162,6 @@ proc `==`*(a, b: PayloadRef): bool =
of RawData: of RawData:
if a.rawBlob != b.rawBlob: if a.rawBlob != b.rawBlob:
return false return false
of RlpData:
if a.rlpBlob != b.rlpBlob:
return false
of AccountData: of AccountData:
if a.account != b.account: if a.account != b.account:
return false return false
@ -219,10 +216,6 @@ func dup*(pld: PayloadRef): PayloadRef =
PayloadRef( PayloadRef(
pType: RawData, pType: RawData,
rawBlob: pld.rawBlob) rawBlob: pld.rawBlob)
of RlpData:
PayloadRef(
pType: RlpData,
rlpBlob: pld.rlpBlob)
of AccountData: of AccountData:
PayloadRef( PayloadRef(
pType: AccountData, pType: AccountData,

243
nimbus/db/aristo/aristo_merge.nim
View File

@ -25,172 +25,121 @@
{.push raises: [].} {.push raises: [].}
import import
std/[strutils, sets, tables, typetraits], std/typetraits,
eth/[common, trie/nibbles], eth/common,
results, results,
"."/[aristo_desc, aristo_get, aristo_hike, aristo_layers, "."/[aristo_desc, aristo_layers, aristo_utils, aristo_vid],
aristo_path, aristo_utils],
./aristo_merge/[merge_payload_helper, merge_proof] ./aristo_merge/[merge_payload_helper, merge_proof]
export export
merge_proof merge_proof
# ------------------------------------------------------------------------------ const
# Private helpers MergeNoAction = {MergeLeafPathCachedAlready, MergeLeafPathOnBackendAlready}
# ------------------------------------------------------------------------------
proc to(
rc: Result[Hike,AristoError];
T: type Result[bool,AristoError];
): T =
## Return code converter
if rc.isOk:
ok true
elif rc.error in {MergeLeafPathCachedAlready,
MergeLeafPathOnBackendAlready}:
ok false
else:
err(rc.error)
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# Public functions # Public functions
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
proc mergePayload*( proc mergeAccountPayload*(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
leafTie: LeafTie; # Leaf item to add to the database accKey: openArray[byte]; # Even nibbled byte path
payload: PayloadRef; # Payload value accPayload: AristoAccount; # Payload value
accPath: PathID; # Needed for accounts payload ): Result[bool,AristoError] =
): Result[Hike,AristoError] = ## Merge the key-value-pair argument `(accKey,accPayload)` as an account
## Merge the argument `leafTie` key-value-pair into the top level vertex ## ledger value, i.e. the the sub-tree starting at `VertexID(1)`.
## table of the database `db`. The field `path` of the `leafTie` argument is
## used to address the leaf vertex with the payload. It is stored or updated
## on the database accordingly.
## ##
## If the `leafTie` argument referes to aa account entrie (i.e. the ## The payload argument `accPayload` must have the `storageID` field either
## `leafTie.root` equals `VertexID(1)`) and the leaf entry has already an ## unset/invalid or referring to a existing vertex which will be assumed
## `AccountData` payload, its `storageID` field must be the same as the one ## to be a storage tree.
## on the database. The `accPath` argument will be ignored.
## ##
## Otherwise, if the `root` argument belongs to a well known sub trie (i.e. let
## it does not exceed `LEAST_FREE_VID`) the `accPath` argument is ignored pyl = PayloadRef(pType: AccountData, account: accPayload)
## and the entry will just be merged. rc = db.mergePayloadImpl(VertexID(1), accKey, pyl, VidVtxPair())
## if rc.isOk:
## Otherwise, a valid `accPath` (i.e. different from `VOID_PATH_ID`.) is ok true
## required relating to an account leaf entry (starting at `VertexID(`)`). elif rc.error in MergeNoAction:
## If the payload of that leaf entry is not of type `AccountData` it is ok false
## ignored.
##
## Otherwise, if the sub-trie where the `leafTie` is to be merged into does
## not exist yes, the `storageID` field of the `accPath` leaf must have been
## reset to `storageID(0)` and will be updated accordingly on the database.
##
## Otherwise its `storageID` field must be equal to the `leafTie.root` vertex
## ID. So vertices can be marked for Merkle hash update.
##
let wp = block:
if leafTie.root.distinctBase < LEAST_FREE_VID:
if not leafTie.root.isValid:
return err(MergeRootMissing)
VidVtxPair()
else:
let rc = db.registerAccount(leafTie.root, accPath)
if rc.isErr:
return err(rc.error)
else:
rc.value
let hike = leafTie.hikeUp(db).to(Hike)
var okHike: Hike
if 0 < hike.legs.len:
case hike.legs[^1].wp.vtx.vType:
of Branch:
okHike = ? db.mergePayloadTopIsBranchAddLeaf(hike, payload)
of Leaf:
if 0 < hike.tail.len: # `Leaf` vertex problem?
return err(MergeLeafGarbledHike)
okHike = ? db.mergePayloadUpdate(hike, leafTie, payload)
of Extension:
okHike = ? db.mergePayloadTopIsExtAddLeaf(hike, payload)
else: else:
# Empty hike err(rc.error)
let rootVtx = db.getVtx hike.root
if rootVtx.isValid:
okHike = ? db.mergePayloadTopIsEmptyAddLeaf(hike,rootVtx, payload) proc mergeGenericData*(
db: AristoDbRef; # Database, top layer
root: VertexID; # MPT state root
path: openArray[byte]; # Leaf item to add to the database
data: openArray[byte]; # Raw data payload value
): Result[bool,AristoError] =
## Variant of `mergeXXX()` for generic sub-trees, i.e. for arguments
## `root` greater than `VertexID(1)` and smaller than `LEAST_FREE_VID`.
##
# Verify that `root` is neither an accounts tree nor a strorage tree.
if not root.isValid:
return err(MergeRootVidMissing)
elif root == VertexID(1):
return err(MergeAccRootNotAccepted)
elif LEAST_FREE_VID <= root.distinctBase:
return err(MergeStoRootNotAccepted)
let
pyl = PayloadRef(pType: RawData, rawBlob: @data)
rc = db.mergePayloadImpl(root, path, pyl, VidVtxPair())
if rc.isOk:
ok true
elif rc.error in MergeNoAction:
ok false
else:
err(rc.error)
proc mergeStorageData*(
db: AristoDbRef; # Database, top layer
stoKey: openArray[byte]; # Storage data path (aka key)
stoData: openArray[byte]; # Storage data payload value
accPath: PathID; # Needed for accounts payload
): Result[VertexID,AristoError] =
## Merge the key-value-pair argument `(stoKey,stoData)` as a storage value.
## This means, the root vertex will be derived from the `accPath` argument,
## the Patricia tree path for the storage tree is given by `stoKey` and the
## leaf value with the payload will be stored as a `PayloadRef` object of
## type `RawData`.
##
## If the storage tree does not exist yet it will be created and the
## payload leaf accessed by `accPath` will be updated with the storage
## tree vertex ID.
##
## The function returns the new vertex ID if a new storage tree was created,
## otherwise `VertexID(0)`.
##
let
wpAcc = ? db.registerAccountForUpdate accPath
stoID = wpAcc.vtx.lData.account.storageID
# Provide new storage ID when needed
useID = if stoID.isValid: stoID else: db.vidFetch()
# Call merge
pyl = PayloadRef(pType: RawData, rawBlob: @stoData)
rc = db.mergePayloadImpl(useID, stoKey, pyl, wpAcc)
if rc.isOk:
if stoID.isValid:
return ok VertexID(0)
else: else:
# Bootstrap for existing root ID # Make sure that there is an account that refers to that storage trie
let wp = VidVtxPair( let leaf = wpAcc.vtx.dup # Dup on modify
vid: hike.root, leaf.lData.account.storageID = useID
vtx: VertexRef( db.layersPutVtx(VertexID(1), wpAcc.vid, leaf)
vType: Leaf, db.layersResKey(VertexID(1), wpAcc.vid)
lPfx: leafTie.path.to(NibblesSeq), return ok useID
lData: payload))
db.setVtxAndKey(hike.root, wp.vid, wp.vtx)
okHike = Hike(root: wp.vid, legs: @[Leg(wp: wp, nibble: -1)])
# Double check the result until the code is more reliable elif rc.error in MergeNoAction:
block: assert stoID.isValid # debugging only
let rc = okHike.to(NibblesSeq).pathToTag return ok VertexID(0)
if rc.isErr or rc.value != leafTie.path:
return err(MergeAssemblyFailed) # Ooops
# Make sure that there is an accounts that refers to that storage trie # else
if wp.vid.isValid and not wp.vtx.lData.account.storageID.isValid: err(rc.error)
let leaf = wp.vtx.dup # Dup on modify
leaf.lData.account.storageID = leafTie.root
db.layersPutVtx(VertexID(1), wp.vid, leaf)
db.layersResKey(VertexID(1), wp.vid)
ok okHike
proc mergePayload*(
db: AristoDbRef; # Database, top layer
root: VertexID; # MPT state root
path: openArray[byte]; # Even nibbled byte path
payload: PayloadRef; # Payload value
accPath = VOID_PATH_ID; # Needed for accounts payload
): Result[bool,AristoError] =
## Variant of `merge()` for `(root,path)` arguments instead of a `LeafTie`
## object.
let lty = LeafTie(root: root, path: ? path.pathToTag)
db.mergePayload(lty, payload, accPath).to(typeof result)
proc merge*(
db: AristoDbRef; # Database, top layer
root: VertexID; # MPT state root
path: openArray[byte]; # Leaf item to add to the database
data: openArray[byte]; # Raw data payload value
accPath: PathID; # Needed for accounts payload
): Result[bool,AristoError] =
## Variant of `merge()` for `(root,path)` arguments instead of a `LeafTie`.
## The argument `data` is stored as-is as a `RawData` payload value.
let pyl = PayloadRef(pType: RawData, rawBlob: @data)
db.mergePayload(root, path, pyl, accPath)
proc mergeAccount*(
db: AristoDbRef; # Database, top layer
path: openArray[byte]; # Leaf item to add to the database
data: openArray[byte]; # Raw data payload value
): Result[bool,AristoError] =
## Variant of `merge()` for `(VertexID(1),path)` arguments instead of a
## `LeafTie`. The argument `data` is stored as-is as a `RawData` payload
## value.
let pyl = PayloadRef(pType: RawData, rawBlob: @data)
db.mergePayload(VertexID(1), path, pyl, VOID_PATH_ID)
proc mergeLeaf*(
db: AristoDbRef; # Database, top layer
leaf: LeafTiePayload; # Leaf item to add to the database
accPath = VOID_PATH_ID; # Needed for accounts payload
): Result[bool,AristoError] =
## Variant of `merge()`. This function will not indicate if the leaf
## was cached, already.
db.mergePayload(leaf.leafTie,leaf.payload, accPath).to(typeof result)
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# End # End

141
nimbus/db/aristo/aristo_merge/merge_payload_helper.nim
View File

@ -11,7 +11,7 @@
{.push raises: [].} {.push raises: [].}
import import
std/[sequtils, sets], std/[sequtils, sets, typetraits],
eth/[common, trie/nibbles], eth/[common, trie/nibbles],
results, results,
".."/[aristo_desc, aristo_get, aristo_hike, aristo_layers, aristo_vid] ".."/[aristo_desc, aristo_get, aristo_hike, aristo_layers, aristo_vid]
@ -33,43 +33,6 @@ proc xPfx(vtx: VertexRef): NibblesSeq =
# Private helpers # Private helpers
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
proc differ(
db: AristoDbRef; # Database, top layer
p1, p2: PayloadRef; # Payload values
): bool =
## Check whether payloads differ on the database.
## If `p1` is `RLP` serialised and `p2` is a raw blob compare serialsations.
## If `p1` is of account type and `p2` is serialised, translate `p2`
## to an account type and compare.
##
if p1 == p2:
return false
# Adjust abd check for divergent types.
if p1.pType != p2.pType:
if p1.pType == AccountData:
try:
let
blob = (if p2.pType == RlpData: p2.rlpBlob else: p2.rawBlob)
acc = rlp.decode(blob, Account)
if acc.nonce == p1.account.nonce and
acc.balance == p1.account.balance and
acc.codeHash == p1.account.codeHash and
acc.storageRoot.isValid == p1.account.storageID.isValid:
if not p1.account.storageID.isValid or
acc.storageRoot.to(HashKey) == db.getKey p1.account.storageID:
return false
except RlpError:
discard
elif p1.pType == RlpData:
if p2.pType == RawData and p1.rlpBlob == p2.rawBlob:
return false
true
# -----------
proc clearMerkleKeys( proc clearMerkleKeys(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
hike: Hike; # Implied vertex IDs to clear hashes for hike: Hike; # Implied vertex IDs to clear hashes for
@ -270,10 +233,10 @@ proc setVtxAndKey*(
db.layersResKey(root, vid) db.layersResKey(root, vid)
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# Public functions: add Particia Trie leaf vertex # Private functions: add Particia Trie leaf vertex
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
proc mergePayloadTopIsBranchAddLeaf*( proc mergePayloadTopIsBranchAddLeaf(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
hike: Hike; # Path top has a `Branch` vertex hike: Hike; # Path top has a `Branch` vertex
payload: PayloadRef; # Leaf data payload payload: PayloadRef; # Leaf data payload
@ -333,7 +296,7 @@ proc mergePayloadTopIsBranchAddLeaf*(
db.insertBranch(hike, linkID, linkVtx, payload) db.insertBranch(hike, linkID, linkVtx, payload)
proc mergePayloadTopIsExtAddLeaf*( proc mergePayloadTopIsExtAddLeaf(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
hike: Hike; # Path top has an `Extension` vertex hike: Hike; # Path top has an `Extension` vertex
payload: PayloadRef; # Leaf data payload payload: PayloadRef; # Leaf data payload
@ -404,7 +367,7 @@ proc mergePayloadTopIsExtAddLeaf*(
ok okHike ok okHike
proc mergePayloadTopIsEmptyAddLeaf*( proc mergePayloadTopIsEmptyAddLeaf(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
hike: Hike; # No path legs hike: Hike; # No path legs
rootVtx: VertexRef; # Root vertex rootVtx: VertexRef; # Root vertex
@ -440,31 +403,25 @@ proc mergePayloadTopIsEmptyAddLeaf*(
db.insertBranch(hike, hike.root, rootVtx, payload) db.insertBranch(hike, hike.root, rootVtx, payload)
# ------------------------------------------------------------------------------
# Public functions
# ------------------------------------------------------------------------------
proc mergePayloadUpdate*( proc mergePayloadUpdate(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
hike: Hike; # No path legs hike: Hike; # Path to payload
leafTie: LeafTie; # Leaf item to add to the database
payload: PayloadRef; # Payload value to add payload: PayloadRef; # Payload value to add
): Result[Hike,AristoError] = ): Result[Hike,AristoError] =
## Update leaf vertex if payloads differ ## Update leaf vertex if payloads differ
let leafLeg = hike.legs[^1] let leafLeg = hike.legs[^1]
# Update payloads if they differ # Update payloads if they differ
if db.differ(leafLeg.wp.vtx.lData, payload): if leafLeg.wp.vtx.lData != payload:
let vid = leafLeg.wp.vid let vid = leafLeg.wp.vid
if vid in db.pPrf: if vid in db.pPrf:
return err(MergeLeafProofModeLock) return err(MergeLeafProofModeLock)
# Verify that the account leaf can be replaced # Make certain that the account leaf can be replaced
if leafTie.root == VertexID(1): if hike.root == VertexID(1):
if leafLeg.wp.vtx.lData.pType != payload.pType: # Only `AccountData` payload on `VertexID(1)` tree
return err(MergeLeafCantChangePayloadType) if payload.account.storageID != leafLeg.wp.vtx.lData.account.storageID:
if payload.pType == AccountData and
payload.account.storageID != leafLeg.wp.vtx.lData.account.storageID:
return err(MergeLeafCantChangeStorageID) return err(MergeLeafCantChangeStorageID)
# Update vertex and hike # Update vertex and hike
@ -486,6 +443,80 @@ proc mergePayloadUpdate*(
else: else:
err(MergeLeafPathCachedAlready) err(MergeLeafPathCachedAlready)
# ------------------------------------------------------------------------------
# Public functions
# ------------------------------------------------------------------------------
proc mergePayloadImpl*(
db: AristoDbRef; # Database, top layer
root: VertexID; # MPT state root
path: openArray[byte]; # Leaf item to add to the database
payload: PayloadRef; # Payload value
wpAcc: VidVtxPair; # Needed for storage tree
): Result[void,AristoError] =
## Merge the argument `(root,path)` key-value-pair into the top level vertex
## table of the database `db`. The `path` argument is used to address the
## leaf vertex with the payload. It is stored or updated on the database
## accordingly.
##
## If the `root` argument is `VertexID(1)` this function relies upon that the
## payload argument is of type `AccountData`. If the payload exists already
## on the database, the `storageID` field of the `payload` and on the database
## must be the same or an error is returned. The argument `wpAcc` will be
## ignored for accounts.
##
## Otherwise, if the `root` argument belongs to a well known sub trie (i.e.
## it does not exceed `LEAST_FREE_VID`) the entry will just be merged. The
## argument `wpAcc` will be ignored .
##
## Otherwise, a valid `wpAcc` must be given referring to an `AccountData`
## payload type leaf vertex. If the `storageID` field of that payload
## does not have a valid entry, a new sub-trie will be created. Otherwise
## this function expects that the `root` argument is the same as the
## `storageID` field.
##
## The function returns `true` iff a new sub-tree was linked to an account
## leaf record.
##
let
nibblesPath = path.initNibbleRange
hike = nibblesPath.hikeUp(root, db).to(Hike)
var okHike: Hike
if 0 < hike.legs.len:
case hike.legs[^1].wp.vtx.vType:
of Branch:
okHike = ? db.mergePayloadTopIsBranchAddLeaf(hike, payload)
of Leaf:
if 0 < hike.tail.len: # `Leaf` vertex problem?
return err(MergeLeafGarbledHike)
okHike = ? db.mergePayloadUpdate(hike, payload)
of Extension:
okHike = ? db.mergePayloadTopIsExtAddLeaf(hike, payload)
else:
# Empty hike
let rootVtx = db.getVtx hike.root
if rootVtx.isValid:
okHike = ? db.mergePayloadTopIsEmptyAddLeaf(hike,rootVtx, payload)
else:
# Bootstrap for existing root ID
let wp = VidVtxPair(
vid: hike.root,
vtx: VertexRef(
vType: Leaf,
lPfx: nibblesPath,
lData: payload))
db.setVtxAndKey(hike.root, wp.vid, wp.vtx)
okHike = Hike(root: wp.vid, legs: @[Leg(wp: wp, nibble: -1)])
# Double check the result (may be removed in future)
if okHike.to(NibblesSeq) != nibblesPath:
return err(MergeAssemblyFailed) # Ooops
ok()
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# End # End
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------

2
nimbus/db/aristo/aristo_serialise.nim
View File

@ -41,8 +41,6 @@ proc serialise(
case pyl.pType: case pyl.pType:
of RawData: of RawData:
ok pyl.rawBlob ok pyl.rawBlob
of RlpData:
ok pyl.rlpBlob
of AccountData: of AccountData:
let let
vid = pyl.account.storageID vid = pyl.account.storageID

2
nimbus/db/aristo/aristo_sign.nim
View File

@ -41,7 +41,7 @@ proc merkleSignAdd*(
## is irrelevant. ## is irrelevant.
if sdb.error == AristoError(0): if sdb.error == AristoError(0):
sdb.count.inc sdb.count.inc
discard sdb.db.merge(sdb.root, key, val, VOID_PATH_ID).valueOr: discard sdb.db.mergeGenericData(sdb.root, key, val).valueOr:
sdb.`error` = error sdb.`error` = error
sdb.errKey = @key sdb.errKey = @key
return return

61
nimbus/db/aristo/aristo_utils.nim
View File

@ -15,7 +15,7 @@
import import
std/[sequtils, sets, typetraits], std/[sequtils, sets, typetraits],
eth/common, eth/[common, trie/nibbles],
results, results,
"."/[aristo_constants, aristo_desc, aristo_get, aristo_hike, aristo_layers] "."/[aristo_constants, aristo_desc, aristo_get, aristo_hike, aristo_layers]
@ -30,13 +30,7 @@ proc toAccount*(
## Converts the argument `payload` to an `Account` type. If the implied ## Converts the argument `payload` to an `Account` type. If the implied
## account das a storage slots system associated, the database `db` must ## account das a storage slots system associated, the database `db` must
## contain the Merkle hash key of the root vertex. ## contain the Merkle hash key of the root vertex.
case payload.pType: if payload.pType == AccountData:
of RlpData:
try:
return ok(rlp.decode(payload.rlpBlob, Account))
except RlpError:
return err(AccRlpDecodingError)
of AccountData:
var acc = Account( var acc = Account(
nonce: payload.account.nonce, nonce: payload.account.nonce,
balance: payload.account.balance, balance: payload.account.balance,
@ -45,8 +39,6 @@ proc toAccount*(
if payload.account.storageID.isValid: if payload.account.storageID.isValid:
acc.storageRoot = (? db.getKeyRc payload.account.storageID).to(Hash256) acc.storageRoot = (? db.getKeyRc payload.account.storageID).to(Hash256)
return ok(acc) return ok(acc)
else:
discard
err PayloadTypeUnsupported err PayloadTypeUnsupported
@ -65,13 +57,7 @@ proc toAccount*(
## Variant of `toAccount()` for a `Leaf` node which must be complete (i.e. ## Variant of `toAccount()` for a `Leaf` node which must be complete (i.e.
## a potential Merkle hash key must have been initialised.) ## a potential Merkle hash key must have been initialised.)
if node.isValid and node.vType == Leaf: if node.isValid and node.vType == Leaf:
case node.lData.pType: if node.lData.pType == AccountData:
of RlpData:
try:
return ok(rlp.decode(node.lData.rlpBlob, Account))
except RlpError:
return err(AccRlpDecodingError)
of AccountData:
var acc = Account( var acc = Account(
nonce: node.lData.account.nonce, nonce: node.lData.account.nonce,
balance: node.lData.account.balance, balance: node.lData.account.balance,
@ -190,7 +176,7 @@ proc registerAccount*(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
stoRoot: VertexID; # Storage root ID stoRoot: VertexID; # Storage root ID
accPath: PathID; # Needed for accounts payload accPath: PathID; # Needed for accounts payload
): Result[VidVtxPair,AristoError] = ): Result[VidVtxPair,AristoError] =
## Verify that the `stoRoot` argument is properly referred to by the ## Verify that the `stoRoot` argument is properly referred to by the
## account data (if any) implied to by the `accPath` argument. ## account data (if any) implied to by the `accPath` argument.
## ##
@ -205,13 +191,14 @@ proc registerAccount*(
# Get account leaf with account data # Get account leaf with account data
let hike = LeafTie(root: VertexID(1), path: accPath).hikeUp(db).valueOr: let hike = LeafTie(root: VertexID(1), path: accPath).hikeUp(db).valueOr:
return err(UtilsAccUnaccessible) return err(UtilsAccInaccessible)
# Check account payload
let wp = hike.legs[^1].wp let wp = hike.legs[^1].wp
if wp.vtx.vType != Leaf: if wp.vtx.vType != Leaf:
return err(UtilsAccPathWithoutLeaf) return err(UtilsAccPathWithoutLeaf)
if wp.vtx.lData.pType != AccountData: if wp.vtx.lData.pType != AccountData:
return ok(VidVtxPair()) # nothing to do return err(UtilsAccLeafPayloadExpected)
# Check whether the `stoRoot` exists on the databse # Check whether the `stoRoot` exists on the databse
let stoVtx = block: let stoVtx = block:
@ -242,6 +229,40 @@ proc registerAccount*(
ok(wp) ok(wp)
proc registerAccountForUpdate*(
db: AristoDbRef; # Database, top layer
accPath: PathID; # Needed for accounts payload
): Result[VidVtxPair,AristoError] =
## ...
##
# Expand vertex path to account leaf
let hike = (@accPath).initNibbleRange.hikeUp(VertexID(1), db).valueOr:
return err(UtilsAccInaccessible)
# Extract the account payload fro the leaf
let wp = hike.legs[^1].wp
if wp.vtx.vType != Leaf:
return err(UtilsAccPathWithoutLeaf)
assert wp.vtx.lData.pType == AccountData # debugging only
let acc = wp.vtx.lData.account
# Check whether storage ID exists, at all
if acc.storageID.isValid:
# Verify that the storage root `acc.storageID` exists on the databse
discard db.getVtxRc(acc.storageID).valueOr:
return err(UtilsStoRootInaccessible)
# Clear Merkle keys so that `hasify()` can calculate the re-hash forest/tree
for w in hike.legs.mapIt(it.wp.vid):
db.layersResKey(hike.root, w)
# Signal to `hashify()` where to start rebuilding Merkel hashes
db.top.final.dirty.incl hike.root
db.top.final.dirty.incl wp.vid
ok(wp)
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# End # End
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------

67
nimbus/db/core_db/backend/aristo_db.nim
View File

@ -18,7 +18,7 @@ import
../../kvt as use_kvt, ../../kvt as use_kvt,
../../kvt/[kvt_init/memory_only, kvt_walk], ../../kvt/[kvt_init/memory_only, kvt_walk],
".."/[base, base/base_desc], ".."/[base, base/base_desc],
./aristo_db/[common_desc, handlers_aristo, handlers_kvt, handlers_trace] ./aristo_db/[common_desc, handlers_aristo, handlers_kvt]
import import
../../aristo/aristo_init/memory_only as aristo_memory_only ../../aristo/aristo_init/memory_only as aristo_memory_only
@ -41,11 +41,11 @@ type
## Main descriptor ## Main descriptor
kdbBase: KvtBaseRef ## Kvt subsystem kdbBase: KvtBaseRef ## Kvt subsystem
adbBase: AristoBaseRef ## Aristo subsystem adbBase: AristoBaseRef ## Aristo subsystem
tracer: AristoTracerRef ## Currently active recorder #tracer: AristoTracerRef ## Currently active recorder
AristoTracerRef = ref object of TraceRecorderRef #AristoTracerRef = ref object of TraceRecorderRef
## Sub-handle for tracer # ## Sub-handle for tracer
parent: AristoCoreDbRef # parent: AristoCoreDbRef
proc newAristoVoidCoreDbRef*(): CoreDbRef {.noRaise.} proc newAristoVoidCoreDbRef*(): CoreDbRef {.noRaise.}
@ -96,28 +96,29 @@ proc txMethods(
raiseAssert info & ": " & $error raiseAssert info & ": " & $error
discard) discard)
proc cptMethods( when false: # currently disabled
tracer: AristoTracerRef; proc cptMethods(
): CoreDbCaptFns = tracer: AristoTracerRef;
let ): CoreDbCaptFns =
tr = tracer # So it can savely be captured let
db = tr.parent # Will not change and can be captured tr = tracer # So it can savely be captured
log = tr.topInst() # Ditto db = tr.parent # Will not change and can be captured
log = tr.topInst() # Ditto
CoreDbCaptFns( CoreDbCaptFns(
recorderFn: proc(): CoreDbRef = recorderFn: proc(): CoreDbRef =
db, db,
logDbFn: proc(): TableRef[Blob,Blob] = logDbFn: proc(): TableRef[Blob,Blob] =
log.kLog, log.kLog,
getFlagsFn: proc(): set[CoreDbCaptFlags] = getFlagsFn: proc(): set[CoreDbCaptFlags] =
log.flags, log.flags,
forgetFn: proc() = forgetFn: proc() =
if not tracer.pop(): if not tracer.pop():
tr.parent.tracer = AristoTracerRef(nil) tr.parent.tracer = AristoTracerRef(nil)
tr.restore()) tr.restore())
proc baseMethods(db: AristoCoreDbRef): CoreDbBaseFns = proc baseMethods(db: AristoCoreDbRef): CoreDbBaseFns =
@ -125,13 +126,14 @@ proc baseMethods(db: AristoCoreDbRef): CoreDbBaseFns =
aBase = db.adbBase aBase = db.adbBase
kBase = db.kdbBase kBase = db.kdbBase
proc tracerSetup(flags: set[CoreDbCaptFlags]): CoreDxCaptRef = when false: # currently disabled
if db.tracer.isNil: proc tracerSetup(flags: set[CoreDbCaptFlags]): CoreDxCaptRef =
db.tracer = AristoTracerRef(parent: db) if db.tracer.isNil:
db.tracer.init(kBase, aBase, flags) db.tracer = AristoTracerRef(parent: db)
else: db.tracer.init(kBase, aBase, flags)
db.tracer.push(flags) else:
CoreDxCaptRef(methods: db.tracer.cptMethods) db.tracer.push(flags)
CoreDxCaptRef(methods: db.tracer.cptMethods)
proc persistent(bn: Opt[BlockNumber]): CoreDbRc[void] = proc persistent(bn: Opt[BlockNumber]): CoreDbRc[void] =
const info = "persistentFn()" const info = "persistentFn()"
@ -182,8 +184,9 @@ proc baseMethods(db: AristoCoreDbRef): CoreDbBaseFns =
dsc = CoreDxTxRef(methods: db.txMethods(aTx, kTx)) dsc = CoreDxTxRef(methods: db.txMethods(aTx, kTx))
db.bless(dsc), db.bless(dsc),
newCaptureFn: proc(flags: set[CoreDbCaptFlags]): CoreDbRc[CoreDxCaptRef] = # # currently disabled
ok(db.bless flags.tracerSetup()), # newCaptureFn: proc(flags:set[CoreDbCaptFlags]): CoreDbRc[CoreDxCaptRef] =
# ok(db.bless flags.tracerSetup()),
persistentFn: proc(bn: Opt[BlockNumber]): CoreDbRc[void] = persistentFn: proc(bn: Opt[BlockNumber]): CoreDbRc[void] =
persistent(bn)) persistent(bn))

3
nimbus/db/core_db/backend/aristo_db/TODO.md Normal file
View File

@ -0,0 +1,3 @@
* Refactor `handlers_tracer`. This module can reliably work only as a genuine
logger. The restore features were ill concieved, an attempt to be as close
as possible to the legacy tracer.

25
nimbus/db/core_db/backend/aristo_db/handlers_aristo.nim
View File

@ -37,7 +37,7 @@ type
AristoCoreDxMptRef = ref object of CoreDxMptRef AristoCoreDxMptRef = ref object of CoreDxMptRef
base: AristoBaseRef ## Local base descriptor base: AristoBaseRef ## Local base descriptor
mptRoot: VertexID ## State root, may be zero unless account mptRoot: VertexID ## State root, may be zero unless account
accPath: PathID ## Needed for storage columns accPath: PathID ## Needed for storage tree/columns
address: EthAddress ## For storage tree debugging address: EthAddress ## For storage tree debugging
AristoColRef* = ref object of CoreDbColRef AristoColRef* = ref object of CoreDbColRef
@ -240,18 +240,17 @@ proc mptMethods(cMpt: AristoCoreDxMptRef): CoreDbMptFns =
proc mptMerge(k: openArray[byte]; v: openArray[byte]): CoreDbRc[void] = proc mptMerge(k: openArray[byte]; v: openArray[byte]): CoreDbRc[void] =
const info = "mergeFn()" const info = "mergeFn()"
# Provide root ID on-the-fly if cMpt.accPath.isValid:
let rootOk = cMpt.mptRoot.isValid let rc = api.mergeStorageData(mpt, k, v, cMpt.accPath)
if not rootOk: if rc.isErr:
cMpt.mptRoot = api.vidFetch(mpt, pristine=true) return err(rc.error.toError(base, info))
if rc.value.isValid:
cMpt.mptRoot = rc.value
else:
let rc = api.mergeGenericData(mpt, cMpt.mptRoot, k, v)
if rc.isErr:
return err(rc.error.toError(base, info))
let rc = api.merge(mpt, cMpt.mptRoot, k, v, cMpt.accPath)
if rc.isErr:
# Re-cycle unused ID (prevents from leaking IDs)
if not rootOk:
api.vidDispose(mpt, cMpt.mptRoot)
cMpt.mptRoot = VoidVID
return err(rc.error.toError(base, info))
ok() ok()
proc mptDelete(key: openArray[byte]): CoreDbRc[void] = proc mptDelete(key: openArray[byte]): CoreDbRc[void] =
@ -350,7 +349,7 @@ proc accMethods(cAcc: AristoCoreDxAccRef): CoreDbAccFns =
let let
key = account.address.keccakHash.data key = account.address.keccakHash.data
val = account.toPayloadRef() val = account.toPayloadRef()
rc = api.mergePayload(mpt, AccountsVID, key, val) rc = api.mergeAccountPayload(mpt, key, val.account)
if rc.isErr: if rc.isErr:
return err(rc.error.toError(base, info)) return err(rc.error.toError(base, info))
ok() ok()

113
nimbus/db/core_db/base.nim
View File

@ -873,66 +873,67 @@ proc dispose*(tx: CoreDxTxRef) =
# Public tracer methods # Public tracer methods
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
proc newCapture*( when false: # currently disabled
db: CoreDbRef; proc newCapture*(
flags: set[CoreDbCaptFlags] = {}; db: CoreDbRef;
): CoreDbRc[CoreDxCaptRef] = flags: set[CoreDbCaptFlags] = {};
## Trace constructor providing an overlay on top of the argument database ): CoreDbRc[CoreDxCaptRef] =
## `db`. This overlay provides a replacement database handle that can be ## Trace constructor providing an overlay on top of the argument database
## retrieved via `db.recorder()` (which can in turn be ovelayed.) While ## `db`. This overlay provides a replacement database handle that can be
## running the overlay stores data in a log-table which can be retrieved ## retrieved via `db.recorder()` (which can in turn be ovelayed.) While
## via `db.logDb()`. ## running the overlay stores data in a log-table which can be retrieved
## ## via `db.logDb()`.
## Caveat: ##
## The original database argument `db` should not be used while the tracer ## Caveat:
## is active (i.e. exists as overlay). The behaviour for this situation ## The original database argument `db` should not be used while the tracer
## is undefined and depends on the backend implementation of the tracer. ## is active (i.e. exists as overlay). The behaviour for this situation
## ## is undefined and depends on the backend implementation of the tracer.
db.setTrackNewApi BaseNewCaptureFn ##
result = db.methods.newCaptureFn flags db.setTrackNewApi BaseNewCaptureFn
db.ifTrackNewApi: debug newApiTxt, api, elapsed, result result = db.methods.newCaptureFn flags
db.ifTrackNewApi: debug newApiTxt, api, elapsed, result
proc recorder*(cpt: CoreDxCaptRef): CoreDbRef = proc recorder*(cpt: CoreDxCaptRef): CoreDbRef =
## Getter, returns a tracer replacement handle to be used as new database. ## Getter, returns a tracer replacement handle to be used as new database.
## It records every action like fetch, store, hasKey, hasPath and delete. ## It records every action like fetch, store, hasKey, hasPath and delete.
## This descriptor can be superseded by a new overlay tracer (using ## This descriptor can be superseded by a new overlay tracer (using
## `newCapture()`, again.) ## `newCapture()`, again.)
## ##
## Caveat: ## Caveat:
## Unless the desriptor `cpt` referes to the top level overlay tracer, the ## Unless the desriptor `cpt` referes to the top level overlay tracer, the
## result is undefined and depends on the backend implementation of the ## result is undefined and depends on the backend implementation of the
## tracer. ## tracer.
## ##
cpt.setTrackNewApi CptRecorderFn cpt.setTrackNewApi CptRecorderFn
result = cpt.methods.recorderFn() result = cpt.methods.recorderFn()
cpt.ifTrackNewApi: debug newApiTxt, api, elapsed cpt.ifTrackNewApi: debug newApiTxt, api, elapsed
proc logDb*(cp: CoreDxCaptRef): TableRef[Blob,Blob] = proc logDb*(cp: CoreDxCaptRef): TableRef[Blob,Blob] =
## Getter, returns the logger table for the overlay tracer database. ## Getter, returns the logger table for the overlay tracer database.
## ##
## Caveat: ## Caveat:
## Unless the desriptor `cpt` referes to the top level overlay tracer, the ## Unless the desriptor `cpt` referes to the top level overlay tracer, the
## result is undefined and depends on the backend implementation of the ## result is undefined and depends on the backend implementation of the
## tracer. ## tracer.
## ##
cp.setTrackNewApi CptLogDbFn cp.setTrackNewApi CptLogDbFn
result = cp.methods.logDbFn() result = cp.methods.logDbFn()
cp.ifTrackNewApi: debug newApiTxt, api, elapsed cp.ifTrackNewApi: debug newApiTxt, api, elapsed
proc flags*(cp: CoreDxCaptRef):set[CoreDbCaptFlags] = proc flags*(cp: CoreDxCaptRef):set[CoreDbCaptFlags] =
## Getter ## Getter
## ##
cp.setTrackNewApi CptFlagsFn cp.setTrackNewApi CptFlagsFn
result = cp.methods.getFlagsFn() result = cp.methods.getFlagsFn()
cp.ifTrackNewApi: debug newApiTxt, api, elapsed, result cp.ifTrackNewApi: debug newApiTxt, api, elapsed, result
proc forget*(cp: CoreDxCaptRef) = proc forget*(cp: CoreDxCaptRef) =
## Explicitely stop recording the current tracer instance and reset to ## Explicitely stop recording the current tracer instance and reset to
## previous level. ## previous level.
## ##
cp.setTrackNewApi CptForgetFn cp.setTrackNewApi CptForgetFn
cp.methods.forgetFn() cp.methods.forgetFn()
cp.ifTrackNewApi: debug newApiTxt, api, elapsed cp.ifTrackNewApi: debug newApiTxt, api, elapsed
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# End # End

15
nimbus/db/core_db/base/validate.nim
View File

@ -38,7 +38,7 @@ proc validateMethodsDesc(base: CoreDbBaseFns) =
doAssert not base.newCtxFromTxFn.isNil doAssert not base.newCtxFromTxFn.isNil
doAssert not base.swapCtxFn.isNil doAssert not base.swapCtxFn.isNil
doAssert not base.beginFn.isNil doAssert not base.beginFn.isNil
doAssert not base.newCaptureFn.isNil # doAssert not base.newCaptureFn.isNil # currently disabled
doAssert not base.persistentFn.isNil doAssert not base.persistentFn.isNil
proc validateMethodsDesc(kvt: CoreDbKvtFns) = proc validateMethodsDesc(kvt: CoreDbKvtFns) =
@ -113,12 +113,13 @@ proc validateMethodsDesc(phk: CoreDxPhkRef) =
doAssert not phk.toMpt.isNil doAssert not phk.toMpt.isNil
phk.methods.validateMethodsDesc phk.methods.validateMethodsDesc
proc validateMethodsDesc(cpt: CoreDxCaptRef) = when false: # currently disabled
doAssert not cpt.isNil proc validateMethodsDesc(cpt: CoreDxCaptRef) =
doAssert not cpt.parent.isNil doAssert not cpt.isNil
doAssert not cpt.methods.recorderFn.isNil doAssert not cpt.parent.isNil
doAssert not cpt.methods.getFlagsFn.isNil doAssert not cpt.methods.recorderFn.isNil
doAssert not cpt.methods.forgetFn.isNil doAssert not cpt.methods.getFlagsFn.isNil
doAssert not cpt.methods.forgetFn.isNil
proc validateMethodsDesc(tx: CoreDxTxRef) = proc validateMethodsDesc(tx: CoreDxTxRef) =
doAssert not tx.isNil doAssert not tx.isNil

2
nimbus/tracer.nim → nimbus/no-tracer.nim
View File

@ -8,6 +8,8 @@
# at your option. This file may not be copied, modified, or distributed except # at your option. This file may not be copied, modified, or distributed except
# according to those terms. # according to those terms.
# TODO: CoreDb module needs to be updated
import import
std/[strutils, json], std/[strutils, json],
./common/common, ./common/common,

7
nimbus/rpc.nim
View File

@ -13,7 +13,7 @@ import
json_rpc/rpcserver, json_rpc/rpcserver,
graphql/httpserver, graphql/httpserver,
./rpc/common, ./rpc/common,
./rpc/debug, #./rpc/debug,
./rpc/engine_api, ./rpc/engine_api,
./rpc/p2p, ./rpc/p2p,
./rpc/jwt_auth, ./rpc/jwt_auth,
@ -59,8 +59,9 @@ proc installRPC(server: RpcServer,
if RpcFlag.Eth in flags: if RpcFlag.Eth in flags:
setupEthRpc(nimbus.ethNode, nimbus.ctx, com, nimbus.txPool, oracle, server) setupEthRpc(nimbus.ethNode, nimbus.ctx, com, nimbus.txPool, oracle, server)
if RpcFlag.Debug in flags: # # Tracer is currently disabled
setupDebugRpc(com, nimbus.txPool, server) # if RpcFlag.Debug in flags:
# setupDebugRpc(com, nimbus.txPool, server)
if RpcFlag.Exp in flags: if RpcFlag.Exp in flags:
setupExpRpc(com, server) setupExpRpc(com, server)

2
tests/all_tests.nim
View File

@ -19,7 +19,7 @@ cliBuilder:
./test_genesis, ./test_genesis,
./test_precompiles, ./test_precompiles,
./test_generalstate_json, ./test_generalstate_json,
./test_tracer_json, #./test_tracer_json, -- temporarily disabled
#./test_persistblock_json, -- fails #./test_persistblock_json, -- fails
#./test_rpc, -- fails #./test_rpc, -- fails
./test_filters, ./test_filters,

53
tests/test_aristo/test_filter.nim
View File

@ -35,6 +35,10 @@ type
DbTriplet = DbTriplet =
array[0..2, AristoDbRef] array[0..2, AristoDbRef]
const
testRootVid = VertexID(2)
## Need to reconfigure for the test, root ID 1 cannot be deleted as a trie
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# Private debugging helpers # Private debugging helpers
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
@ -74,7 +78,7 @@ iterator quadripartite(td: openArray[ProofTrieData]): LeafQuartet =
var collect: seq[seq[LeafTiePayload]] var collect: seq[seq[LeafTiePayload]]
for w in td: for w in td:
let lst = w.kvpLst.mapRootVid VertexID(1) let lst = w.kvpLst.mapRootVid testRootVid
if lst.len < 8: if lst.len < 8:
if 2 < collect.len: if 2 < collect.len:
@ -101,40 +105,53 @@ proc dbTriplet(w: LeafQuartet; rdbPath: string): Result[DbTriplet,AristoError] =
let db = block: let db = block:
if 0 < rdbPath.len: if 0 < rdbPath.len:
let rc = AristoDbRef.init(RdbBackendRef, rdbPath, DbOptions.init()) let rc = AristoDbRef.init(RdbBackendRef, rdbPath, DbOptions.init())
xCheckRc rc.error == 0 xCheckRc rc.error == 0:
result = err(rc.error)
rc.value rc.value
else: else:
AristoDbRef.init MemBackendRef AristoDbRef.init MemBackendRef
block:
# Add a dummy entry so the balancer logic can be triggered in `persist()`
let rc = db.mergeDummyAccLeaf(0, 0)
xCheckRc rc.error == 0:
result = err(rc.error)
# Set failed `xCheck()` error result
result = err(AristoError 1)
# Fill backend # Fill backend
block: block:
let report = db.mergeList w[0] let report = db.mergeList w[0]
if report.error != 0: if report.error != 0:
db.finish(eradicate=true) db.finish(eradicate=true)
check report.error == 0 xCheck report.error == 0
return err(report.error)
let rc = db.persist() let rc = db.persist()
if rc.isErr: xCheckRc rc.error == 0:
check rc.error == 0 result = err(rc.error)
return
let dx = [db, db.forkTx(0).value, db.forkTx(0).value] let dx = [db, db.forkTx(0).value, db.forkTx(0).value]
xCheck dx[0].nForked == 2 xCheck dx[0].nForked == 2
# Reduce unwanted tx layers # Reduce unwanted tx layers
for n in 1 ..< dx.len: for n in 1 ..< dx.len:
check dx[n].level == 1 xCheck dx[n].level == 1
check dx[n].txTop.value.commit.isOk xCheck dx[n].txTop.value.commit.isOk
# Clause (9) from `aristo/README.md` example # Clause (9) from `aristo/README.md` example
for n in 0 ..< dx.len: for n in 0 ..< dx.len:
let report = dx[n].mergeList w[n+1] let report = dx[n].mergeList w[n+1]
if report.error != 0: if report.error != 0:
db.finish(eradicate=true) db.finish(eradicate=true)
check (n, report.error) == (n,0) xCheck (n, report.error) == (n,0)
return err(report.error)
ok dx block:
# Add a dummy entry so the balancer logic can be triggered in `persist()`
let rc = db.mergeDummyAccLeaf(0, 1)
xCheckRc rc.error == 0:
result = err(rc.error)
return ok(dx)
# ---------------------- # ----------------------
@ -152,7 +169,7 @@ proc isDbEq(a, b: LayerDeltaRef; db: AristoDbRef; noisy = true): bool =
return false return false
if unsafeAddr(a[]) != unsafeAddr(b[]): if unsafeAddr(a[]) != unsafeAddr(b[]):
if a.src != b.src or if a.src != b.src or
a.kMap.getOrVoid(VertexID 1) != b.kMap.getOrVoid(VertexID 1) or a.kMap.getOrVoid(testRootVid) != b.kMap.getOrVoid(testRootVid) or
a.vTop != b.vTop: a.vTop != b.vTop:
return false return false
@ -280,6 +297,11 @@ proc testDistributedAccess*(
xCheck db1.balancer == LayerDeltaRef(nil) xCheck db1.balancer == LayerDeltaRef(nil)
xCheck db2.balancer == db3.balancer xCheck db2.balancer == db3.balancer
block:
# Add dummy entry so the balancer logic can be triggered in `persist()`
let rc = db2.mergeDummyAccLeaf(n, 42)
xCheckRc rc.error == 0
block: block:
let rc = db2.stow() # non-persistent let rc = db2.stow() # non-persistent
xCheckRc rc.error == 0: xCheckRc rc.error == 0:
@ -320,6 +342,11 @@ proc testDistributedAccess*(
defer: defer:
dy.cleanUp() dy.cleanUp()
block:
# Add dummy entry so the balancer logic can be triggered in `persist()`
let rc = db2.mergeDummyAccLeaf(n, 42)
xCheckRc rc.error == 0
# Build clause (12) from `aristo/README.md` example # Build clause (12) from `aristo/README.md` example
discard db2.reCentre() discard db2.reCentre()
block: block:

62
tests/test_aristo/test_helpers.nim
View File

@ -11,7 +11,7 @@
import import
std/[os, sequtils], std/[os, sequtils],
eth/common, eth/common,
rocksdb, stew/endians2,
../../nimbus/db/aristo/[ ../../nimbus/db/aristo/[
aristo_debug, aristo_desc, aristo_delete, aristo_debug, aristo_desc, aristo_delete,
aristo_hashify, aristo_hike, aristo_merge], aristo_hashify, aristo_hike, aristo_merge],
@ -247,31 +247,13 @@ proc delTree*(
aristo_delete.delTree(db, root, accPath) aristo_delete.delTree(db, root, accPath)
proc merge*( proc mergeGenericData*(
db: AristoDbRef; db: AristoDbRef; # Database, top layer
root: VertexID; leaf: LeafTiePayload; # Leaf item to add to the database
path: openArray[byte]; ): Result[bool,AristoError] =
data: openArray[byte]; ## Variant of `mergeGenericData()`.
accPath: PathID; db.mergeGenericData(
noisy: bool; leaf.leafTie.root, @(leaf.leafTie.path), leaf.payload.rawBlob)
): Result[bool, AristoError] =
when declared(aristo_merge.noisy):
aristo_merge.exec(noisy, aristo_merge.merge(db, root, path, data, accPath))
else:
aristo_merge.merge(db, root, path, data, accPath)
proc mergePayload*(
db: AristoDbRef;
lty: LeafTie;
pyl: PayloadRef;
accPath: PathID;
noisy: bool;
): Result[Hike,AristoError] =
when declared(aristo_merge.noisy):
aristo_merge.exec(noisy, aristo_merge.mergePayload(db, lty, pyl, accPath))
else:
aristo_merge.mergePayload(db, lty, pyl, accPath)
proc mergeList*( proc mergeList*(
db: AristoDbRef; # Database, top layer db: AristoDbRef; # Database, top layer
@ -283,20 +265,36 @@ proc mergeList*(
for n,w in leafs: for n,w in leafs:
noisy.say "*** mergeList", noisy.say "*** mergeList",
" n=", n, "/", leafs.len " n=", n, "/", leafs.len
let rc = db.mergePayload(w.leafTie, w.payload, VOID_PATH_ID, noisy=noisy) let rc = db.mergeGenericData w
noisy.say "*** mergeList", noisy.say "*** mergeList",
" n=", n, "/", leafs.len, " n=", n, "/", leafs.len,
" rc=", (if rc.isOk: "ok" else: $rc.error), " rc=", (if rc.isOk: "ok" else: $rc.error),
"\n -------------\n" "\n -------------\n"
if rc.isOk: if rc.isErr:
merged.inc
elif rc.error in {MergeLeafPathCachedAlready,MergeLeafPathOnBackendAlready}:
dups.inc
else:
return (n,dups,rc.error) return (n,dups,rc.error)
elif rc.value:
merged.inc
else:
dups.inc
(merged, dups, AristoError(0)) (merged, dups, AristoError(0))
proc mergeDummyAccLeaf*(
db: AristoDbRef;
pathID: int;
nonce: int;
): Result[void,AristoError] =
# Add a dummy entry so the balancer logic can be triggered
let
acc = AristoAccount(nonce: nonce.AccountNonce)
rc = db.mergeAccountPayload(pathID.uint64.toBytesBE, acc)
if rc.isOk:
ok()
else:
err(rc.error)
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# End # End
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------

83
tests/test_aristo/test_tx.nim
View File

@ -43,6 +43,9 @@ const
MaxFilterBulk = 150_000 MaxFilterBulk = 150_000
## Policy settig for `pack()` ## Policy settig for `pack()`
testRootVid = VertexID(2)
## Need to reconfigure for the test, root ID 1 cannot be deleted as a trie
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# Private helpers # Private helpers
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
@ -306,25 +309,6 @@ proc revWalkVerify(
true true
proc mergeRlpData*(
db: AristoDbRef; # Database, top layer
path: PathID; # Path into database
rlpData: openArray[byte]; # RLP encoded payload data
): Result[void,AristoError] =
block body:
discard db.mergeLeaf(
LeafTiePayload(
leafTie: LeafTie(
root: VertexID(1),
path: path.normal),
payload: PayloadRef(
pType: RlpData,
rlpBlob: @rlpData))).valueOr:
if error in {MergeLeafPathCachedAlready,MergeLeafPathOnBackendAlready}:
break body
return err(error)
ok()
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
# Public test function # Public test function
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
@ -361,14 +345,14 @@ proc testTxMergeAndDeleteOneByOne*(
# Reset database so that the next round has a clean setup # Reset database so that the next round has a clean setup
defer: db.innerCleanUp defer: db.innerCleanUp
# Merge leaf data into main trie (w/vertex ID 1) # Merge leaf data into main trie
let kvpLeafs = block: let kvpLeafs = block:
var lst = w.kvpLst.mapRootVid VertexID(1) var lst = w.kvpLst.mapRootVid testRootVid
# The list might be reduced for isolation of particular properties, # The list might be reduced for isolation of particular properties,
# e.g. lst.setLen(min(5,lst.len)) # e.g. lst.setLen(min(5,lst.len))
lst lst
for i,leaf in kvpLeafs: for i,leaf in kvpLeafs:
let rc = db.mergeLeaf leaf let rc = db.mergeGenericData leaf
xCheckRc rc.error == 0 xCheckRc rc.error == 0
# List of all leaf entries that should be on the database # List of all leaf entries that should be on the database
@ -394,14 +378,17 @@ proc testTxMergeAndDeleteOneByOne*(
doSaveBeOk = ((u mod saveMod) == saveRest) doSaveBeOk = ((u mod saveMod) == saveRest)
(leaf, lid) = lvp (leaf, lid) = lvp
# Add a dummy entry so the balancer logic can be triggered
let rc = db.mergeDummyAccLeaf(n, runID)
xCheckRc rc.error == 0
if doSaveBeOk: if doSaveBeOk:
let saveBeOk = tx.saveToBackend( let saveBeOk = tx.saveToBackend(
chunkedMpt=false, relax=relax, noisy=noisy, runID) chunkedMpt=false, relax=relax, noisy=noisy, runID)
xCheck saveBeOk: xCheck saveBeOk:
noisy.say "***", "del(2)", noisy.say "***", "del1by1(2)",
" u=", u, " u=", u,
" n=", n, "/", list.len, " n=", n, "/", list.len,
"\n leaf=", leaf.pp(db),
"\n db\n ", db.pp(backendOk=true), "\n db\n ", db.pp(backendOk=true),
"" ""
@ -414,7 +401,8 @@ proc testTxMergeAndDeleteOneByOne*(
leafsLeft.excl leaf leafsLeft.excl leaf
let deletedVtx = tx.db.getVtx lid let deletedVtx = tx.db.getVtx lid
xCheck deletedVtx.isValid == false xCheck deletedVtx.isValid == false:
noisy.say "***", "del1by1(8)"
# Walking the database is too slow for large tables. So the hope is that # Walking the database is too slow for large tables. So the hope is that
# potential errors will not go away and rather pop up later, as well. # potential errors will not go away and rather pop up later, as well.
@ -430,7 +418,9 @@ proc testTxMergeAndDeleteOneByOne*(
return return
when true and false: when true and false:
noisy.say "***", "del(9) n=", n, "/", list.len, " nLeafs=", kvpLeafs.len noisy.say "***", "del1by1(9)",
" n=", n, "/", list.len,
" nLeafs=", kvpLeafs.len
true true
@ -440,9 +430,6 @@ proc testTxMergeAndDeleteSubTree*(
list: openArray[ProofTrieData]; list: openArray[ProofTrieData];
rdbPath: string; # Rocks DB storage directory rdbPath: string; # Rocks DB storage directory
): bool = ): bool =
const
# Need to reconfigure for the test, root ID 1 cannot be deleted as a trie
testRootVid = VertexID(2)
var var
prng = PrngDesc.init 42 prng = PrngDesc.init 42
db = AristoDbRef(nil) db = AristoDbRef(nil)
@ -460,9 +447,10 @@ proc testTxMergeAndDeleteSubTree*(
else: else:
AristoDbRef.init(MemBackendRef) AristoDbRef.init(MemBackendRef)
if testRootVid != VertexID(1): # Add a dummy entry so the balancer logic can be triggered
# Add a dummy entry so the journal logic can be triggered block:
discard db.merge(VertexID(1), @[n.byte], @[42.byte], VOID_PATH_ID) let rc = db.mergeDummyAccLeaf(n, 42)
xCheckRc rc.error == 0
# Start transaction (double frame for testing) # Start transaction (double frame for testing)
xCheck db.txTop.isErr xCheck db.txTop.isErr
@ -480,7 +468,7 @@ proc testTxMergeAndDeleteSubTree*(
# e.g. lst.setLen(min(5,lst.len)) # e.g. lst.setLen(min(5,lst.len))
lst lst
for i,leaf in kvpLeafs: for i,leaf in kvpLeafs:
let rc = db.mergeLeaf leaf let rc = db.mergeGenericData leaf
xCheckRc rc.error == 0 xCheckRc rc.error == 0
# List of all leaf entries that should be on the database # List of all leaf entries that should be on the database
@ -511,9 +499,10 @@ proc testTxMergeAndDeleteSubTree*(
"\n db\n ", db.pp(backendOk=true), "\n db\n ", db.pp(backendOk=true),
"" ""
if testRootVid != VertexID(1): # Update dummy entry so the journal logic can be triggered
# Update dummy entry so the journal logic can be triggered block:
discard db.merge(VertexID(1), @[n.byte], @[43.byte], VOID_PATH_ID) let rc = db.mergeDummyAccLeaf(n, 43)
xCheckRc rc.error == 0
block: block:
let saveBeOk = tx.saveToBackend( let saveBeOk = tx.saveToBackend(
@ -571,15 +560,22 @@ proc testTxMergeProofAndKvpList*(
count = 0 count = 0
count.inc count.inc
# Add a dummy entry so the balancer logic can be triggered
block:
let rc = db.mergeDummyAccLeaf(n, 42)
xCheckRc rc.error == 0
let let
testId = idPfx & "#" & $w.id & "." & $n testId = idPfx & "#" & $w.id & "." & $n
runID = n runID = n
sTabLen = db.nLayersVtx() sTabLen = db.nLayersVtx()
leafs = w.kvpLst.mapRootVid VertexID(1) # merge into main trie leafs = w.kvpLst.mapRootVid testRootVid # merge into main trie
# var lst = w.kvpLst.mapRootVid testRootVid
if 0 < w.proof.len: if 0 < w.proof.len:
let root = block: let root = block:
let rc = db.mergeProof(rootKey, VertexID(1)) let rc = db.mergeProof(rootKey, testRootVid)
xCheckRc rc.error == 0 xCheckRc rc.error == 0
rc.value rc.value
@ -596,13 +592,14 @@ proc testTxMergeProofAndKvpList*(
xCheck merged.merged + merged.dups == leafs.len xCheck merged.merged + merged.dups == leafs.len
block: block:
let oops = oopsTab.getOrDefault(testId,(0,AristoError(0))) let
if not tx.saveToBackendWithOops( oops = oopsTab.getOrDefault(testId,(0,AristoError(0)))
chunkedMpt=true, noisy=noisy, debugID=runID, oops): saveBeOk = tx.saveToBackendWithOops(
return chunkedMpt=true, noisy=noisy, debugID=runID, oops)
xCheck saveBeOk
when true and false: when true and false:
noisy.say "***", "testTxMergeProofAndKvpList (1)", noisy.say "***", "testTxMergeProofAndKvpList (9)",
" <", n, "/", list.len-1, ">", " <", n, "/", list.len-1, ">",
" runID=", runID, " runID=", runID,
" groups=", count, " merged=", merged " groups=", count, " merged=", merged

18
tests/test_tools_build.nim
View File

@ -12,18 +12,18 @@
{. warning[UnusedImport]:off .} {. warning[UnusedImport]:off .}
import import
../premix/premix, #../premix/premix, # -- currently disabled (no tracer at the moment)
../premix/persist, #../premix/persist, # -- ditto
../premix/debug, #../premix/debug, # -- ditto
../premix/dumper, #../premix/dumper, # -- ditto
../premix/hunter, #../premix/hunter, # -- ditto
../premix/regress, #../premix/regress, # -- ditto
./tracerTestGen, #./tracerTestGen, # -- ditto
./persistBlockTestGen, #./persistBlockTestGen, # -- ditto
../hive_integration/nodocker/rpc/rpc_sim, ../hive_integration/nodocker/rpc/rpc_sim,
../hive_integration/nodocker/consensus/consensus_sim, ../hive_integration/nodocker/consensus/consensus_sim,
../hive_integration/nodocker/graphql/graphql_sim, ../hive_integration/nodocker/graphql/graphql_sim,
../hive_integration/nodocker/engine/engine_sim, #../hive_integration/nodocker/engine/engine_sim, # -- does not compile
../hive_integration/nodocker/pyspec/pyspec_sim, ../hive_integration/nodocker/pyspec/pyspec_sim,
../tools/t8n/t8n, ../tools/t8n/t8n,
../tools/t8n/t8n_test, ../tools/t8n/t8n_test,