* fix fuzzer so that it doesn't do exact comparison - this doesn't work
any more because the implementations no longer match byte-for-byte -
instead, we check that the libraries agree on valid/invalid and that C++
can decompress snappy-encoded data
Some data is already protected by stronger checks - crc32 on the other
hand significantly slows down framed reading - ie 2.5x slower:
```
118.853 / 41.781, 129.115 / 0.000, 188.438 / 0.000, 90.565 / 44.371, 50, 115613038, state-6800000-488b7150-d613b584.ssz
186.600 / 97.202, 191.935 /123.325, 0.000 / 0.000, 0.000 / 0.000, 50, 115613038, state-6800000-488b7150-d613b584.ssz(framed)
```
The difference between unframed and framed decoding is the CRC32 check -
it takes ~50ms on a decent laptop for a 110mb file.
This is a more or less complete revamp of the snappy library aiming to:
* clear out a lot of the duplicate code
* remove some of the redundant API
* unify the codebase behind a single, optimized "inner" encoder/decoder
* unify the public API for in-memory and stream
compression/decompression
* improve performance
As such, only the documented API remains backwards-compatible - the rest
has been refactored, moved around and rewritten:
* `import snappy` now exposes only in-memory encoders / decoders
* framed format moved to `snappy` module, `snappy/framing` removed
* faststreams integration moved to `snappy/faststreams`
* minimal `std/streams` integration started in `snappy/streams`
Other changes include:
* up-to-date documentation
* allocation- and exception-free API (uses some amount of stack memory)
* a 2-3x improvement to both compression and decompression performance,
putting the library mostly on par with the C++ implementation (see
README)
* the implementation was heavily inspired by the `C++`, `C` and `go`
implementations, but somewhat simplified
* nonetheless, the code uses a significant amount of unsafe code to
work around inefficiencies in the safe subset of Nim
With bulk operations in place, the cost of range checks falls
significantly - we can reintroduce them without any significant loss in
performance by carefully ordering operations such that optimizers can
elide most.
The snappy codebase is a mess with competing implementations,
nonsensical code duplication and no real direction due to a partially
implemented faststreams migration.
This PR makes it slightly less of a mess, but make no mistake, it's
still a mess - the difference being that there are a few more signposts
along the way in terms of module organisation, and a little less mess as
the line count of the PR discloses.
Performance remains poor - ~3x slower than C++ - but at least there's
less code to look at :)
The `appendSnappyBytes` implementation of `snappy` computes indices
incorrectly, resulting in wrong data being produced. The implementation
was fixed and the test suite extended accordingly. Note that this issue
is not reachable because `appendSnappyBytes` is only used in test code.
Etan Kissling
Jacek Sieka
jangko
Zahary Karadjov