* fix fuzzer so that it doesn't do exact comparison - this doesn't work
any more because the implementations no longer match byte-for-byte -
instead, we check that the libraries agree on valid/invalid and that C++
can decompress snappy-encoded data
Some data is already protected by stronger checks - crc32 on the other
hand significantly slows down framed reading - ie 2.5x slower:
```
118.853 / 41.781, 129.115 / 0.000, 188.438 / 0.000, 90.565 / 44.371, 50, 115613038, state-6800000-488b7150-d613b584.ssz
186.600 / 97.202, 191.935 /123.325, 0.000 / 0.000, 0.000 / 0.000, 50, 115613038, state-6800000-488b7150-d613b584.ssz(framed)
```
The difference between unframed and framed decoding is the CRC32 check -
it takes ~50ms on a decent laptop for a 110mb file.
This is a more or less complete revamp of the snappy library aiming to:
* clear out a lot of the duplicate code
* remove some of the redundant API
* unify the codebase behind a single, optimized "inner" encoder/decoder
* unify the public API for in-memory and stream
compression/decompression
* improve performance
As such, only the documented API remains backwards-compatible - the rest
has been refactored, moved around and rewritten:
* `import snappy` now exposes only in-memory encoders / decoders
* framed format moved to `snappy` module, `snappy/framing` removed
* faststreams integration moved to `snappy/faststreams`
* minimal `std/streams` integration started in `snappy/streams`
Other changes include:
* up-to-date documentation
* allocation- and exception-free API (uses some amount of stack memory)
* a 2-3x improvement to both compression and decompression performance,
putting the library mostly on par with the C++ implementation (see
README)
* the implementation was heavily inspired by the `C++`, `C` and `go`
implementations, but somewhat simplified
* nonetheless, the code uses a significant amount of unsafe code to
work around inefficiencies in the safe subset of Nim
With bulk operations in place, the cost of range checks falls
significantly - we can reintroduce them without any significant loss in
performance by carefully ordering operations such that optimizers can
elide most.
The snappy codebase is a mess with competing implementations,
nonsensical code duplication and no real direction due to a partially
implemented faststreams migration.
This PR makes it slightly less of a mess, but make no mistake, it's
still a mess - the difference being that there are a few more signposts
along the way in terms of module organisation, and a little less mess as
the line count of the PR discloses.
Performance remains poor - ~3x slower than C++ - but at least there's
less code to look at :)
When using `framingFormatCompress`, the given payload is compressed to
determine whether its compressed form is shorter than its raw form.
For short payloads the Snappy compression will never be shorter, so it
is not necessary to compress such payloads. Instead, short payloads can
always be treated as uncompressable data. This patch optimizes for that.
Miran
Zahary Karadjov
Jacek Sieka
Etan Kissling
jangko
Zahary Karadjov