Pure Nim implementation of deflate, zlib, gzip and zip.
Go to file
jangko 8d6828f090
add gc:orc and gc:arc test in .nimble
2021-05-28 10:15:16 +07:00
.github/workflows change github action setup to use nimbus-build-system 2021-05-28 10:00:16 +07:00
examples example updated 2021-03-02 18:12:11 -06:00
src 0.5.7 2021-05-02 20:15:46 -05:00
tests change github action setup to use nimbus-build-system 2021-05-28 10:00:16 +07:00
.gitattributes add .gitattributes to prevent automatic EOL conversion 2021-05-28 09:14:57 +07:00
.gitignore tarballs 2020-12-21 16:10:18 -06:00
LICENSE license 2020-10-13 20:05:19 -05:00
README.md change github action setup to use nimbus-build-system 2021-05-28 10:00:16 +07:00
zippy.nimble add gc:orc and gc:arc test in .nimble 2021-05-28 10:15:16 +07:00

README.md

Zippy

Github action

nimble install zippy

Zippy is an implementation of DEFLATE, ZLIB and GZIP data compression formats. Zippy can also create and open Tarballs (.tar, .tar.gz, .tgz, .taz) and ZIP archives.

The goal of this library is to be a pure Nim implementation that is small, performant and dependency-free.

Zippy can also be used at compile time. This is great for baking assets into executables in compressed form. Check out an example here.

To ensure Zippy is compatible with other implementations, tests/validate.nim can be run. This script verifies that data compressed by Zippy can be uncompressed by other implementations (and that other implementations can uncompress data compressed by Zippy).

This library works well using Nim's relatively new --gc:arc and --gc:orc as well as the default garbage collector. This library also works using both nim c and nim cpp, in addition to --cc:vcc on Windows.

I have also verified that Zippy builds with --experimental:strictFuncs on Nim 1.4.0.

Examples

Simple examples using Zippy can be found in the examples/ folder.

Performance

Benchmarks can be run comparing different deflate implementations. My benchmarking shows this library performs very well, a bit faster than zlib in some cases and a bit slower in others. Check the performance yourself by running tests/benchmark.nim.

nim c -d:release -r .\tests\benchmark.nim

The times below are measured on a Ryzen 5 5600X.

Compress

Each file is compressed 10 times per run.

Default compression

https://github.com/guzba/zippy results:

file min avg std dv runs % red
alice29.txt 29.543 ms 30.698 ms ±0.188 x167 63.32%
urls.10K 157.261 ms 166.513 ms ±0.449 x31 67.49%
rfctest3.gold 5.486 ms 5.621 ms ±0.079 x973 70.73%
randtest3.gold 0.757 ms 0.777 ms ±0.015 x1000 0%
paper-100k.pdf 15.460 ms 16.046 ms ±0.109 x331 19.94%
geo.protodata 10.012 ms 10.434 ms ±0.068 x522 86.91%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file min avg std dv runs % red
alice29.txt 58.013 ms 58.213 ms ±0.177 x86 64.23%
urls.10K 132.910 ms 133.379 ms ±0.380 x38 68.29%
rfctest3.gold 6.525 ms 6.589 ms ±0.060 x757 71.74%
randtest3.gold 0.887 ms 0.903 ms ±0.015 x1000 0%
paper-100k.pdf 15.020 ms 15.113 ms ±0.058 x331 20.59%
geo.protodata 8.996 ms 9.094 ms ±0.065 x549 87.24%

Fastest compression

https://github.com/guzba/zippy results:

file min avg std dv runs % red
alice29.txt 12.745 ms 13.261 ms ±0.153 x405 55.32%
urls.10K 40.832 ms 41.720 ms ±0.131 x127 61.70%
rfctest3.gold 3.430 ms 3.522 ms ±0.022 x1000 66.31%
randtest3.gold 0.291 ms 0.306 ms ±0.009 x1000 0%
paper-100k.pdf 9.154 ms 9.399 ms ±0.049 x600 18.44%
geo.protodata 6.609 ms 6.747 ms ±0.016 x844 80.42%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file min avg std dv runs % red
alice29.txt 12.797 ms 13.024 ms ±0.198 x384 57.17%
urls.10K 53.304 ms 53.515 ms ±0.122 x94 63.93%
rfctest3.gold 2.179 ms 2.220 ms ±0.029 x1000 67.53%
randtest3.gold 0.813 ms 0.831 ms ±0.016 x1000 0%
paper-100k.pdf 12.806 ms 12.878 ms ±0.056 x388 20.22%
geo.protodata 3.494 ms 3.531 ms ±0.024 x1000 84.12%

Best compression

https://github.com/guzba/zippy results:

file min avg std dv runs % red
alice29.txt 36.293 ms 37.635 ms ±0.149 x136 63.75%
urls.10K 229.110 ms 244.351 ms ±0.431 x22 68.14%
rfctest3.gold 10.681 ms 11.173 ms ±0.121 x465 70.92%
randtest3.gold 0.762 ms 0.780 ms ±0.008 x1000 0%
paper-100k.pdf 16.124 ms 16.833 ms ±0.078 x311 20.07%
geo.protodata 10.795 ms 11.222 ms ±0.046 x459 87.07%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file min avg std dv runs % red
alice29.txt 86.051 ms 86.287 ms ±0.216 x58 64.38%
urls.10K 253.127 ms 253.952 ms ±0.724 x20 68.82%
rfctest3.gold 21.753 ms 22.018 ms ±0.168 x227 71.77%
randtest3.gold 0.887 ms 0.902 ms ±0.014 x1000 0%
paper-100k.pdf 16.749 ms 16.838 ms ±0.056 x297 20.64%
geo.protodata 12.252 ms 12.346 ms ±0.066 x405 87.37%

Uncompress

Each file is uncompressed 10 times per run.

https://github.com/guzba/zippy results:

file min avg std dv runs
alice29.txt.z 3.570 ms 3.595 ms ±0.018 x1000
urls.10K.z 17.324 ms 17.392 ms ±0.057 x288
rfctest3.z 0.777 ms 0.794 ms ±0.011 x1000
randtest3.z 0.067 ms 0.070 ms ±0.006 x1000
paper-100k.pdf.z 2.908 ms 2.937 ms ±0.034 x1000
geo.protodata.z 1.268 ms 1.299 ms ±0.016 x1000

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file min avg std dv runs
alice29.txt.z 3.530 ms 3.572 ms ±0.016 x1000
urls.10K.z 15.029 ms 15.093 ms ±0.053 x331
rfctest3.z 0.452 ms 0.481 ms ±0.037 x1000
randtest3.z 0.065 ms 0.072 ms ±0.011 x1000
paper-100k.pdf.z 2.208 ms 2.221 ms ±0.009 x1000
geo.protodata.z 0.897 ms 0.918 ms ±0.016 x1000

Testing

nimble test

To prevent Zippy from causing a crash or otherwise misbehaving on bad input data, a fuzzer has been run against it. You can do run the fuzzer any time by running nim c -r tests/fuzz.nim

API: zippy

import zippy

const NoCompression

NoCompression = 0

const BestSpeed

BestSpeed = 1

const BestCompression

BestCompression = 9

const DefaultCompression

DefaultCompression = -1

const HuffmanOnly

HuffmanOnly = -2

type CompressedDataFormat

Supported compressed data formats

CompressedDataFormat = enum
 dfDetect, dfZlib, dfGzip, dfDeflate

func compress

Compresses src and returns the compressed data.

func compress(src: seq[uint8]; level = DefaultCompression; dataFormat = dfGzip): seq[
 uint8] {.raises: [ZippyError].}

template compress

Helper for when preferring to work with strings.

template compress(src: string; level = DefaultCompression; dataFormat = dfGzip): string

func uncompress

Uncompresses src and returns the uncompressed data seq.

func uncompress(src: seq[uint8]; dataFormat = dfDetect): seq[uint8] {.raises: [ZippyError].}

template uncompress

Helper for when preferring to work with strings.

template uncompress(src: string; dataFormat = dfDetect): string

type ZippyError

Raised if an operation fails.

ZippyError = object of ValueError

API: zippy/tarballs

import zippy/tarballs

type EntryKind

EntryKind = enum
 ekNormalFile = 48, ekDirectory = 53

type TarballEntry

TarballEntry = object
 kind*: EntryKind
 contents*: string
 lastModified*: times.Time

type Tarball

Tarball = ref object
 contents*: OrderedTable[string, TarballEntry]

proc addDir

Recursively adds all of the files and directories inside dir to tarball.

proc addDir(tarball: Tarball; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}

proc open

Opens the tarball file located at path and reads its contents into tarball.contents (clears any existing tarball.contents entries). Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc open(tarball: Tarball; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}

proc writeTarball

Writes tarball.contents to a tarball file at path. Uses the path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc writeTarball(tarball: Tarball; path: string) {.raises: [ZippyError, IOError], tags: [WriteIOEffect].}

proc extractAll

Extracts the files stored in tarball to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).

proc extractAll(tarball: Tarball; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}

proc extractAll

Extracts the files in the tarball located at tarPath into the destination directory. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc extractAll(tarPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

proc createTarball

Creates a tarball containing all of the files and directories inside source and writes the tarball file to dest. Uses the dest path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc createTarball(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}

API: zippy/ziparchives

import zippy/ziparchives

type EntryKind

EntryKind = enum
 ekFile, ekDirectory

type ArchiveEntry

ArchiveEntry = object
 kind*: EntryKind
 contents*: string

type ZipArchive

ZipArchive = ref object
 contents*: OrderedTable[string, ArchiveEntry]

proc addDir

Recursively adds all of the files and directories inside dir to archive.

proc addDir(archive: ZipArchive; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}

proc open

Opens the zip archive file located at path and reads its contents into archive.contents (clears any existing archive.contents entries).

proc open(archive: ZipArchive; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}

proc writeZipArchive

Writes archive.contents to a zip file at path.

proc writeZipArchive(archive: ZipArchive; path: string) {.raises: [ZippyError, ZippyError, IOError], tags: [WriteIOEffect].}

proc extractAll

Extracts the files stored in archive to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).

proc extractAll(archive: ZipArchive; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}

proc extractAll

Extracts the files in the archive located at zipPath into the destination directory.

proc extractAll(zipPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

proc createZipArchive

Creates an archive containing all of the files and directories inside source and writes the zip file to dest.

proc createZipArchive(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}