mirror of https://github.com/status-im/leopard.git
94 lines
3.0 KiB
Markdown
94 lines
3.0 KiB
Markdown
# Leopard-RS
|
|
## Leopard Reed-Solomon Error Correction Codes in C
|
|
|
|
#### This software is still under active development. It may or may not work right now. I'm trying to get it done ASAP. Current latest result is that K=128 code rate 1/2 is working and benchmarks are posted here: [http://catid.mechafetus.com/news/news.php?view=399](http://catid.mechafetus.com/news/news.php?view=399)
|
|
|
|
Leopard-RS is a portable, fast library for Forward Error Correction.
|
|
From a block of equally sized original data pieces, it generates recovery
|
|
symbols that can be used to recover lost original data.
|
|
|
|
* It requires that data pieces are all a fixed size, a multiple of 64 bytes.
|
|
* The original and recovery data must not exceed 65536 pieces.
|
|
|
|
|
|
#### Motivation:
|
|
|
|
It gets slower as O(N Log N) in the input data size, and its inner loops are
|
|
vectorized using the best approaches available on modern processors, using the
|
|
fastest finite fields (8-bit or 16-bit Galois fields) for bulk data.
|
|
|
|
It sets new speed records for MDS encoding and decoding of large data.
|
|
It is also the only open-source production ready software for this purpose
|
|
available today.
|
|
|
|
Example applications are data recovery software and data center replication.
|
|
|
|
|
|
#### Encoder API:
|
|
|
|
```
|
|
#include "leopard.h"
|
|
```
|
|
|
|
For full documentation please read `leopard.h`.
|
|
|
|
+ `leo_init()` : Initialize library.
|
|
+ `leo_encode_work_count()` : Calculate the number of work_data buffers to provide to leo_encode().
|
|
+ `leo_encode()`: Generate recovery data.
|
|
|
|
|
|
#### Decoder API:
|
|
|
|
```
|
|
#include "leopard.h"
|
|
```
|
|
|
|
For full documentation please read `leopard.h`.
|
|
|
|
+ `leo_init()` : Initialize library.
|
|
+ `leo_decode_work_count()` : Calculate the number of work_data buffers to provide to leo_decode().
|
|
+ `leo_decode()` : Generate recovery data.
|
|
|
|
|
|
#### Benchmarks:
|
|
|
|
```
|
|
TODO
|
|
```
|
|
|
|
|
|
#### Comparisons:
|
|
|
|
```
|
|
TODO
|
|
```
|
|
|
|
|
|
#### Background
|
|
|
|
This library implements an MDS erasure code introduced in this paper:
|
|
|
|
~~~
|
|
S.-J. Lin, T. Y. Al-Naffouri, Y. S. Han, and W.-H. Chung,
|
|
"Novel Polynomial Basis with Fast Fourier Transform and Its Application to Reed-Solomon Erasure Codes"
|
|
IEEE Trans. on Information Theory, pp. 6284-6299, November, 2016.
|
|
~~~
|
|
|
|
The paper is available here: [http://ct.ee.ntust.edu.tw/it2016-2.pdf](http://ct.ee.ntust.edu.tw/it2016-2.pdf)
|
|
And also mirrored in the /docs/ folder.
|
|
|
|
The high-level summary is that instead of using complicated fields,
|
|
an additive FFT was introduced that works with familiar Galois fields for the first time.
|
|
This is actually a huge new result that will change how Reed-Solomon codecs will be written.
|
|
|
|
My contribution is extending the ALTMAP approach from Jerasure
|
|
for 16-bit Galois fields out to 64 bytes to enable AVX2 speedups,
|
|
and marry it with the row parallelism introduced by ISA-L.
|
|
|
|
|
|
#### Credits
|
|
|
|
The idea is the brain-child of S.-J. Lin. He is a super bright guy who should be recognized more widely!
|
|
|
|
This software was written entirely by myself ( Christopher A. Taylor mrcatid@gmail.com ). If you find it useful and would like to buy me a coffee, consider tipping.
|