status-im
/
nim-snappy
mirror of https://github.com/status-im/nim-snappy.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
66 Commits 6 Branches 0 Tags 1.9 MiB
Commit Graph

66 Commits

Author SHA1 Message Date
narimiran c196893450
move -d:nimRawSetjmp to nim.cfg 
Refs https://github.com/status-im/nimbus-build-system/issues/44
 2022-06-17 19:08:25 +02:00
Jacek Sieka 7cb2e57a58
snappy revamp (#10) 
This is a more or less complete revamp of the snappy library aiming to:

* clear out a lot of the duplicate code
* remove some of the redundant API
* unify the codebase behind a single, optimized "inner" encoder/decoder
* unify the public API for in-memory and stream
compression/decompression
* improve performance

As such, only the documented API remains backwards-compatible - the rest
has been refactored, moved around and rewritten:

* `import snappy` now exposes only in-memory encoders / decoders
* framed format moved to `snappy` module, `snappy/framing` removed
* faststreams integration moved to `snappy/faststreams`
* minimal `std/streams` integration started in `snappy/streams`

Other changes include:

* up-to-date documentation
* allocation- and exception-free API (uses some amount of stack memory)
* a 2-3x improvement to both compression and decompression performance,
putting the library mostly on par with the C++ implementation (see
README)
* the implementation was heavily inspired by the `C++`, `C` and `go`
implementations, but somewhat simplified
* nonetheless, the code uses a significant amount of unsafe code to
work around inefficiencies in the safe subset of Nim

With bulk operations in place, the cost of range checks falls
significantly - we can reintroduce them without any significant loss in
performance by carefully ordering operations such that optimizers can
elide most.
 2022-04-14 16:22:41 +02:00
Jacek Sieka 16bf7b7d96
deduplicate and reorganise code (#9) 
The snappy codebase is a mess with competing implementations,
nonsensical code duplication and no real direction due to a partially
implemented faststreams migration.

This PR makes it slightly less of a mess, but make no mistake, it's
still a mess - the difference being that there are a few more signposts
along the way in terms of module organisation, and a little less mess as
the line count of the PR discloses.

Performance remains poor - ~3x slower than C++ - but at least there's
less code to look at :)
 2022-04-01 12:57:39 +02:00
Jacek Sieka 6537b10600
Merge pull request #8 from status-im/raises-framed 
Raises framed
 2022-03-29 09:09:31 +02:00
Jacek Sieka 6afa8377e1
raises annotations for framed format 2022-03-28 14:08:06 +02:00
Ștefan Talpalaru 3d39a6228a
CI: test with multiple Nim versions (#7) 
* CI: test with multiple Nim versions
 2022-01-11 20:25:13 +01:00
Jacek Sieka 1b3f8d60a8
Merge pull request #24 from status-im/snappy-refresh 
bump cppsnappy, split out benchmark
 2022-01-02 19:05:54 +01:00
Jacek Sieka 2256d6efb2
ensure benchmarks are built on test 2021-12-30 13:54:55 +01:00
Jacek Sieka 45f2d5d84a
bump cppsnappy, split out benchmark 
* benchmark both compression and decompression
* bump C++-snappy: it's gotten better over the years

```
fastStreams,       openArrays,       nimStreams,           cppLib,      Samples,         Size,         Test
  0.281 /  0.139,   0.295 /  0.220,   0.528 /  0.238,   0.140 /  0.047,          100,       102400, html
  2.697 /  1.457,   3.263 /  1.993,   6.106 /  2.026,   1.655 /  0.530,          100,       702087, urls.10K
  0.021 /  0.011,   0.017 /  0.011,   0.031 /  0.011,   0.014 /  0.009,          100,       123093, fireworks.jpeg
  0.057 /  0.019,   0.051 /  0.036,   0.092 /  0.042,   0.021 /  0.011,          100,       102400, paper-100k.pdf
  1.153 /  0.565,   1.210 /  0.904,   2.172 /  0.985,   0.575 /  0.184,          100,       409600, html_x_4
  0.887 /  0.582,   0.976 /  0.743,   1.923 /  0.712,   0.536 /  0.202,          100,       152089, alice29.txt
  0.771 /  0.504,   0.850 /  0.639,   1.678 /  0.606,   0.482 /  0.184,          100,       129301, asyoulik.txt
  2.349 /  1.518,   2.553 /  1.946,   5.057 /  1.913,   1.391 /  0.522,          100,       426754, lcet10.txt
  2.983 /  1.951,   3.241 /  2.408,   6.680 /  2.342,   1.882 /  0.735,          100,       481861, plrabn12.txt
  0.293 /  0.120,   0.306 /  0.210,   0.542 /  0.241,   0.131 /  0.042,          100,       118588, geo.protodata
  0.743 /  0.501,   0.738 /  0.620,   1.597 /  0.640,   0.414 /  0.191,          100,       184320, kppkn.gtb
  0.087 /  0.054,   0.102 /  0.067,   0.193 /  0.061,   0.052 /  0.022,          100,        14564, Mark.Twain-Tom.Sawyer.txt
 66.886 / 20.880, 105.393 / 37.907, 210.197 / 38.867,  34.273 / 10.193,           10,     38942424, state-2560000-114a593d-0d5e08e8.ssz
```

In general, we're consistently about 2x slower than C++ right now.
 2021-12-30 13:33:27 +01:00
jangko 7f51d29126
windows-ci: bump mingw gcc from v8.1.0 to v11.2.0 2021-12-14 23:31:07 +07:00
Etan Kissling 16cce7d07c fix `appendSnappyBytes` index computation 
The `appendSnappyBytes` implementation of `snappy` computes indices
incorrectly, resulting in wrong data being produced. The implementation
was fixed and the test suite extended accordingly. Note that this issue
is not reachable because `appendSnappyBytes` is only used in test code.
 2021-12-14 23:19:24 +07:00
Etan Kissling d555230013 avoid unnecessary compression of short payloads 
When using `framingFormatCompress`, the given payload is compressed to
determine whether its compressed form is shorter than its raw form.
For short payloads the Snappy compression will never be shorter, so it
is not necessary to compress such payloads. Instead, short payloads can
always be treated as uncompressable data. This patch optimizes for that.
 2021-12-14 23:18:34 +07:00
Ștefan Talpalaru 5750797ded
CI: refactor Nim compiler caching 2021-06-01 04:05:29 +02:00
Jacek Sieka be86aed2ad
Merge pull request #19 from status-im/fix_fishy_code 
fixes fishy and dubious codes
 2021-02-17 13:30:38 +01:00
jangko 22dbb2eb65
fixes fishy and dubious codes 2021-02-03 19:17:19 +07:00
andri lim 4c50008ab2
Merge pull request #17 from status-im/github_action 
add github action
 2020-12-22 16:00:33 +07:00
jangko ab5bbf624a
add github action script 2020-12-22 15:27:06 +07:00
andri lim b9b3d4931c
Merge pull request #16 from status-im/snappycpp_submodule 
submoduling snappycpp
 2020-12-22 12:53:04 +07:00
jangko 6861fc1aae
fixes ci script 2020-12-22 12:34:07 +07:00
jangko b10f16da7a
submoduling snappycpp 2020-12-22 12:17:39 +07:00
Jacek Sieka 5a8166b786 use stew/leb128 2020-12-15 17:07:58 +02:00
Jacek Sieka 8455b825e5
Merge pull request #13 from status-im/silly-operator 
fix unnecessary seq allocation
 2020-08-26 18:46:47 +02:00
Jacek Sieka 07cea69de5
fix unnecessary seq allocation 
this significantly slows down the implementation
 2020-08-26 15:52:15 +02:00
Zahary Karadjov 1e506c80a9
Remove some unused code 2020-08-19 14:19:51 +03:00
Zahary Karadjov 6da2be2564
Fix a logical typo 2020-08-19 13:26:18 +03:00
Zahary Karadjov a368549c1a
Fix various integer overflow issues found through fuzzing 2020-08-18 23:11:42 +03:00
Zahary Karadjov f449a5a47a
Allow bounding the maximum decoded size 2020-08-18 17:34:55 +03:00
Zahary Karadjov cae6c07fbb
Fix the fuzzing test 2020-07-22 20:25:22 +03:00
andri lim 676fa656d3
Merge pull request #10 from status-im/fix_crash 
fix crc32c nim side crash
 2020-05-24 09:36:00 +07:00
jangko 700f7777fd
fix crc32c nim side crash 2020-05-23 13:21:55 +07:00
andri lim 672ecf54d9
Merge pull request #8 from status-im/fuzzing-tests 
Added fuzzing tests
 2020-05-21 12:03:10 +07:00
jangko 9e1856da95
fix appveyor.yml: ignore warning when building snappycpp instead of treating it as errors 2020-05-21 11:50:08 +07:00
Zahary Karadjov f6a87764a3
Added fuzzing tests 2020-05-20 21:06:13 +03:00
Zahary Karadjov b4cd68e27a
Add a helper for framing compression of blobs 2020-05-12 22:30:21 +03:00
Zahary Karadjov 20cc8ce1c2
Export more constants, so they can be used in NBC 2020-05-08 22:24:16 +03:00
Zahary Karadjov 5e9e2a1f65
Async version of the Snappy framing format based on the latest FastStreams version 2020-05-06 00:35:55 +03:00
Zahary Karadjov b1726ab7e7
Add a helper for one-shot framing-format decompression 2020-04-16 14:59:42 +03:00
Zahary Karadjov 80cff583e3
More faststrams upgrades; Re-enable the file-based tests 2020-04-14 17:00:40 +03:00
Zahary Karadjov b5196c17b6
Use the latest faststreams OutputStream API 2020-04-13 15:02:41 +03:00
andri lim f08cbf9dc5 working snappy framing compress prototype 2020-04-01 22:35:57 +03:00
andri lim 45b8258af4 renormalize *.txt EOL 2020-04-01 22:35:57 +03:00
andri lim 73bb7db070 working framing uncompress prototype 2020-04-01 22:35:57 +03:00
andri lim 71b24a6d15 add framing format test runner 2020-04-01 22:35:57 +03:00
andri lim bb20886854 implement crc stuff 2020-04-01 22:35:57 +03:00
andri lim 6722c7431e add nim-stew dependency 2020-04-01 22:35:57 +03:00
andri lim 791df10be7 fixes ci 2020-04-01 22:35:57 +03:00
andri lim 4dc604a5ca add framing format spec file 2020-04-01 22:35:57 +03:00
andri lim aafdb7c8e6 move maxEncodedLen to snappy/utils.nim 2020-04-01 22:35:57 +03:00
andri lim 1bfb3800f1 modify .gitignore 2020-04-01 22:35:57 +03:00
andri lim 68857f7b14 remove not needed import 2019-09-03 12:32:26 +03:00