Commit Graph

3 Commits

Author SHA1 Message Date
Jacek Sieka ecbcee1d10
allow skipping crc32 integrity check (#22)
Some data is already protected by stronger checks - crc32 on the other
hand significantly slows down framed reading - ie 2.5x slower:

```
118.853 / 41.781, 129.115 /  0.000, 188.438 /  0.000,  90.565 / 44.371,           50,    115613038, state-6800000-488b7150-d613b584.ssz
186.600 / 97.202, 191.935 /123.325,   0.000 /  0.000,   0.000 /  0.000,           50,    115613038, state-6800000-488b7150-d613b584.ssz(framed)
```

The difference between unframed and framed decoding is the CRC32 check -
it takes ~50ms on a decent laptop for a 110mb file.
2023-07-25 18:50:36 +03:00
Jacek Sieka e36f19d886
clean up Defect (#21) 2023-07-21 14:44:16 +02:00
Jacek Sieka 7cb2e57a58
snappy revamp (#10)
This is a more or less complete revamp of the snappy library aiming to:

* clear out a lot of the duplicate code
* remove some of the redundant API
* unify the codebase behind a single, optimized "inner" encoder/decoder
* unify the public API for in-memory and stream
compression/decompression
* improve performance

As such, only the documented API remains backwards-compatible - the rest
has been refactored, moved around and rewritten:

* `import snappy` now exposes only in-memory encoders / decoders
* framed format moved to `snappy` module, `snappy/framing` removed
* faststreams integration moved to `snappy/faststreams`
* minimal `std/streams` integration started in `snappy/streams`

Other changes include:

* up-to-date documentation
* allocation- and exception-free API (uses some amount of stack memory)
* a 2-3x improvement to both compression and decompression performance,
putting the library mostly on par with the C++ implementation (see
README)
* the implementation was heavily inspired by the `C++`, `C` and `go`
implementations, but somewhat simplified
* nonetheless, the code uses a significant amount of unsafe code to
work around inefficiencies in the safe subset of Nim

With bulk operations in place, the cost of range checks falls
significantly - we can reintroduce them without any significant loss in
performance by carefully ordering operations such that optimizers can
elide most.
2022-04-14 16:22:41 +02:00