rebalance rocksdb cache sizes (#2557)

Based on some simple testing done with a few combinations of cache
sizes, it seems that the block cache has grown in importance compared to
the where we were before changing on-disk format and adding a lot of
other point caches.

With these settings, there's roughly a 15% performance increase when
processing blocks in the 18M range over the status quo while memory
usage decreases by more than 1gb!

Only a few values were tested so there's certainly more to do here but
this change sets up a better baseline for any future optimizations.

In particular, since the initial defaults were chosen root vertex id:s
were introduced as key prefixes meaning that storage for each account
will be grouped together and thus it becomes more likely that a block
loaded from disk will be hit multiple times - this seems to give the
block cache an edge over the row cache, specially when traversing the
storage trie.

nimbus/db/core_db/backend/aristo_rocksdb.nim

@@ -49,7 +49,13 @@ proc toRocksDb*(
   tableOpts.filterPolicy = createRibbonHybrid(9.9)
 
   if opts.blockCacheSize > 0:
-    # Share a single block cache instance between all column families
+    # The block cache holds uncompressed data blocks that each contain multiple
+    # key-value pairs - it helps in particular when loading sort-adjacent values
+    # such as when the storage of each account is prefixed by a value unique to
+    # that account - it is best that this cache is large enough to hold a
+    # significant portion of the inner trie nodes!
+    # This code sets up a single block cache to be shared, a strategy that
+    # plausibly can be refined in the future.
     tableOpts.blockCache = cacheCreateLRU(opts.blockCacheSize, autoClose = true)
 
   # Single-level indices might cause long stalls due to their large size -
@@ -114,9 +120,10 @@ proc toRocksDb*(
   dbOpts.maxOpenFiles = opts.maxOpenFiles
 
   if opts.rowCacheSize > 0:
-    # Good for GET queries, which is what we do most of the time - if we start
-    # using range queries, we should probably give more attention to the block
-    # cache
+    # Good for GET queries, which is what we do most of the time - however,
+    # because we have other similar caches at different abstraction levels in
+    # the codebase, this cache ends up being less impactful than the block cache
+    # even though it is faster to access.
     # https://github.com/facebook/rocksdb/blob/af50823069818fc127438e39fef91d2486d6e76c/include/rocksdb/options.h#L1276
     dbOpts.rowCache = cacheCreateLRU(opts.rowCacheSize, autoClose = true)
 

nimbus/db/opts.nim

@@ -18,8 +18,8 @@ const
   # https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning
   defaultMaxOpenFiles* = 512
   defaultWriteBufferSize* = 64 * 1024 * 1024
-  defaultRowCacheSize* = 4096 * 1024 * 1024
-  defaultBlockCacheSize* = 256 * 1024 * 1024
+  defaultRowCacheSize* = 1024 * 1024 * 1024
+  defaultBlockCacheSize* = 2 * 1024 * 1024 * 1024
 
 type DbOptions* = object # Options that are transported to the database layer
   maxOpenFiles*: int