132 lines
6.3 KiB
Markdown
132 lines
6.3 KiB
Markdown
# Snappy
|
|
[](https://travis-ci.org/status-im/nim-snappy)
|
|
[](https://ci.appveyor.com/project/nimbus/nim-snappy/branch/master)
|
|
|
|
|
|
|
|
|
|
Compression and decompression utilities for the `snappy` compression algorithm:
|
|
|
|
* [Overview](http://google.github.io/snappy/)
|
|
* [Format description](https://github.com/google/snappy/blob/main/format_description.txt)
|
|
|
|
The main module, `snappy`, contains in-memory encoders and decoders:
|
|
|
|
* `compress`/`uncompress` work with caller-allocated buffers
|
|
* No dynamic memory allocation (the functions require ~20kb stack space)
|
|
* Exception-free
|
|
* `encode`/`decode` are convenience wrappers for the above that take care of
|
|
memory allocation
|
|
* Simplified error reporting
|
|
* Suitable for small buffers mainly
|
|
|
|
Framed encodings are also supported via functions carrying the `Framed` suffix.
|
|
|
|
* [Framing format](https://github.com/google/snappy/blob/main/framing_format.txt)
|
|
|
|
## Stream support
|
|
|
|
The library supports compression and decompression for the following libraries
|
|
|
|
* [faststreams](https://github.com/status-im/nim-faststreams)
|
|
* `import snappy/faststreams`
|
|
* [std/streams](https://nim-lang.org/docs/streams.html)
|
|
* `import snappy/streams` (incomplete)
|
|
|
|
## API
|
|
|
|
### In-memory
|
|
|
|
```nim
|
|
import snappy
|
|
|
|
func compress*(
|
|
input: openArray[byte],
|
|
output: var openArray[byte]): Result[int, CodecError]
|
|
func encode*(input: openArray[byte]): seq[byte]
|
|
func uncompress*(input: openArray[byte], output: var openArray[byte]):
|
|
Result[int, CodecError]
|
|
func decode*(input: openArray[byte], maxSize = maxUncompressedLen): seq[byte]
|
|
```
|
|
|
|
### faststreams
|
|
|
|
:warning: BETA API, subject to change
|
|
|
|
When using faststreams, errors are reported via exceptions.
|
|
|
|
Uncompressing raw snappy is not covered in streaming mode due to the requirement that full uncompressed data must be available during decompression.
|
|
|
|
```nim
|
|
import snappy/faststreams
|
|
|
|
proc compress*(input: InputStream, output: OutputStream)
|
|
proc compressFramed*(input: InputStream, output: OutputStream)
|
|
proc uncompressFramed*(input: InputStream, output: OutputStream)
|
|
```
|
|
|
|
### std/streams
|
|
|
|
:warning: BETA API, subject to change
|
|
|
|
```nim
|
|
import snappy/streams
|
|
|
|
proc compress*(input: Stream, inputLen: int, output: Stream)
|
|
# TODO compressFramed
|
|
# TODO uncompressFramed
|
|
```
|
|
|
|
## Examples
|
|
```Nim
|
|
import snappy
|
|
var source = readFile("readme.md")
|
|
var encoded = snappy.encode(toOpenArrayByte(source, 0, source.len-1))
|
|
var decoded = snappy.decode(encoded)
|
|
assert equalMem(decoded[0].addr, source[0].addr, source.len)
|
|
```
|
|
|
|
## Performance
|
|
|
|
Generally, performance is on par with the C++ implementation, shown as `cppLib`.
|
|
|
|
Framed encoding is slower due to the extra CRC32C processing.
|
|
|
|
The table shows average time to compress data in `ms` on x86_64. Lower is better.
|
|
|
|
```
|
|
inMemory, fastStreams, nimStreams, cppLib, Samples, Size, Test
|
|
0.086 / 0.056, 0.087 / 0.000, 0.112 / 0.000, 0.088 / 0.029, 100, 102400, html
|
|
0.117 / 0.093, 0.118 / 0.094, 0.000 / 0.000, 0.000 / 0.000, 100, 102400, html(framed)
|
|
1.052 / 0.480, 1.073 / 0.000, 1.322 / 0.000, 1.005 / 0.335, 100, 702087, urls.10K
|
|
1.260 / 0.775, 1.286 / 0.785, 0.000 / 0.000, 0.000 / 0.000, 100, 702087, urls.10K(framed)
|
|
0.008 / 0.005, 0.022 / 0.000, 0.092 / 0.000, 0.008 / 0.005, 100, 123093, fireworks.jpeg
|
|
0.051 / 0.047, 0.067 / 0.057, 0.000 / 0.000, 0.000 / 0.000, 100, 123093, fireworks.jpeg(framed)
|
|
0.010 / 0.006, 0.021 / 0.000, 0.066 / 0.000, 0.009 / 0.005, 100, 102400, paper-100k.pdf
|
|
0.046 / 0.050, 0.057 / 0.054, 0.000 / 0.000, 0.000 / 0.000, 100, 102400, paper-100k.pdf(framed)
|
|
0.374 / 0.218, 0.378 / 0.000, 0.451 / 0.000, 0.357 / 0.118, 100, 409600, html_x_4
|
|
0.491 / 0.386, 0.498 / 0.392, 0.000 / 0.000, 0.000 / 0.000, 100, 409600, html_x_4(framed)
|
|
0.334 / 0.186, 0.345 / 0.000, 0.399 / 0.000, 0.331 / 0.126, 100, 152089, alice29.txt
|
|
0.382 / 0.251, 0.392 / 0.251, 0.000 / 0.000, 0.000 / 0.000, 100, 152089, alice29.txt(framed)
|
|
0.300 / 0.165, 0.311 / 0.000, 0.354 / 0.000, 0.300 / 0.114, 100, 129301, asyoulik.txt
|
|
0.343 / 0.220, 0.352 / 0.222, 0.000 / 0.000, 0.000 / 0.000, 100, 129301, asyoulik.txt(framed)
|
|
0.907 / 0.483, 0.932 / 0.000, 1.086 / 0.000, 0.880 / 0.327, 100, 426754, lcet10.txt
|
|
1.053 / 0.675, 1.075 / 0.680, 0.000 / 0.000, 0.000 / 0.000, 100, 426754, lcet10.txt(framed)
|
|
1.241 / 0.646, 1.272 / 0.000, 1.477 / 0.000, 1.201 / 0.466, 100, 481861, plrabn12.txt
|
|
1.387 / 0.856, 1.425 / 0.861, 0.000 / 0.000, 0.000 / 0.000, 100, 481861, plrabn12.txt(framed)
|
|
0.076 / 0.050, 0.075 / 0.000, 0.096 / 0.000, 0.076 / 0.025, 100, 118588, geo.protodata
|
|
0.110 / 0.095, 0.112 / 0.098, 0.000 / 0.000, 0.000 / 0.000, 100, 118588, geo.protodata(framed)
|
|
0.279 / 0.183, 0.287 / 0.000, 0.338 / 0.000, 0.273 / 0.121, 100, 184320, kppkn.gtb
|
|
0.346 / 0.261, 0.354 / 0.263, 0.000 / 0.000, 0.000 / 0.000, 100, 184320, kppkn.gtb(framed)
|
|
0.024 / 0.018, 0.026 / 0.000, 0.032 / 0.000, 0.024 / 0.014, 100, 14564, Mark.Twain-Tom.Sawyer.txt
|
|
0.030 / 0.021, 0.031 / 0.021, 0.000 / 0.000, 0.000 / 0.000, 100, 14564, Mark.Twain-Tom.Sawyer.txt(framed)
|
|
23.814 / 8.608, 27.362 / 0.000, 48.342 / 0.000, 22.157 / 6.958, 50, 38942424, state-2560000-114a593d-0d5e08e8.ssz
|
|
36.075 / 25.389, 39.979 / 28.497, 0.000 / 0.000, 0.000 / 0.000, 50, 38942424, state-2560000-114a593d-0d5e08e8.ssz(framed)```
|
|
```
|
|
|
|
## Installation via nimble
|
|
|
|
```bash
|
|
nimble install snappy
|
|
```
|