status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Use zstd compression in bottommost layer (#2582) 

Tested up to block ~14m, zstd uses ~12% less space which seems to result
in a small:ish (2-4%) performance improvement on block import speed -
this seems like a better baseline for more extensive testing in the
future.

Pre: 57383308 kb
Post: 50831236 kb
This commit is contained in:
Jacek Sieka 2024-08-30 17:32:13 +02:00 committed by GitHub
parent 42a08cfba9
commit 84a72c8658
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 7 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
nimbus/db/core_db/backend/aristo_rocksdb.nim
View File

@ -93,13 +93,16 @@ proc toRocksDb*(
cfOpts.memtableWholeKeyFiltering = true cfOpts.memtableWholeKeyFiltering = true
cfOpts.memtablePrefixBloomSizeRatio = 0.1 cfOpts.memtablePrefixBloomSizeRatio = 0.1
# LZ4 seems to cut database size to 2/3 roughly, at the time of writing # ZSTD seems to cut database size to 2/3 roughly, at the time of writing
# Using it for the bottom-most level means it applies to 90% of data but # Using it for the bottom-most level means it applies to 90% of data but
# delays compression until data has settled a bit, which seems like a # delays compression until data has settled a bit, which seems like a
# reasonable tradeoff. # reasonable tradeoff.
# TODO evaluate zstd compression with a trained dictionary # Compared to LZ4 that was tested earlier, the default ZSTD config results
# https://github.com/facebook/rocksdb/wiki/Compression # in 10% less space and similar or slightly better performance in some
cfOpts.bottommostCompression = Compression.lz4Compression # simple tests around mainnet block 14M.
# TODO evaluate zstd dictionary compression
# https://github.com/facebook/rocksdb/wiki/Dictionary-Compression
cfOpts.bottommostCompression = Compression.zstdCompression
# TODO In the AriVtx table, we don't do lookups that are expected to result # TODO In the AriVtx table, we don't do lookups that are expected to result
# in misses thus we could avoid the filter cost - this does not apply to # in misses thus we could avoid the filter cost - this does not apply to