Use zstd compression in bottommost layer (#2582)

Tested up to block ~14m, zstd uses ~12% less space which seems to result
in a small:ish (2-4%) performance improvement on block import speed -
this seems like a better baseline for more extensive testing in the
future.

Pre: 57383308 kb
Post: 50831236 kb
This commit is contained in:
Jacek Sieka 2024-08-30 17:32:13 +02:00 committed by GitHub
parent 42a08cfba9
commit 84a72c8658
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 7 additions and 4 deletions

View File

@ -93,13 +93,16 @@ proc toRocksDb*(
cfOpts.memtableWholeKeyFiltering = true
cfOpts.memtablePrefixBloomSizeRatio = 0.1
# LZ4 seems to cut database size to 2/3 roughly, at the time of writing
# ZSTD seems to cut database size to 2/3 roughly, at the time of writing
# Using it for the bottom-most level means it applies to 90% of data but
# delays compression until data has settled a bit, which seems like a
# reasonable tradeoff.
# TODO evaluate zstd compression with a trained dictionary
# https://github.com/facebook/rocksdb/wiki/Compression
cfOpts.bottommostCompression = Compression.lz4Compression
# Compared to LZ4 that was tested earlier, the default ZSTD config results
# in 10% less space and similar or slightly better performance in some
# simple tests around mainnet block 14M.
# TODO evaluate zstd dictionary compression
# https://github.com/facebook/rocksdb/wiki/Dictionary-Compression
cfOpts.bottommostCompression = Compression.zstdCompression
# TODO In the AriVtx table, we don't do lookups that are expected to result
# in misses thus we could avoid the filter cost - this does not apply to