codex-storage
/
das-research
mirror of https://github.com/codex-storage/das-research.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
238 Commits 79 Branches 0 Tags 2.6 MiB
Commit Graph

81 Commits

Author SHA1 Message Date
Csaba Kiraly 08e67cbeb1
measure validation progress more precisely 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-05-02 12:19:08 +02:00
Csaba Kiraly 1638a18507
make failureRate exact 
simplify code and make sure failureRate is the exact
portion of segments missing, not just a probability per
sample.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-04-18 14:54:38 +02:00
Csaba Kiraly 7e4074938a
add duplicate statistics 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-27 22:29:22 +02:00
Csaba Kiraly 894009b414
Validator node: add nodeClass property 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-27 11:37:22 +02:00
Csaba Kiraly ff93161b8d
set vpn in validator 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-27 11:37:22 +02:00
Csaba Kiraly 6e4b37a3d2
adding log level TRACE 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-27 11:37:22 +02:00
Csaba Kiraly 353efec141
fixup: use vpn1 and vpn2 correctly 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 20:38:58 +01:00
Csaba Kiraly c366c05616
handle overlap for multiple validators per node correctly 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 20:38:57 +01:00
Csaba Kiraly 9d9612fd34
rename numberValidators to numberNodes 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 20:38:53 +01:00
Csaba Kiraly 065086f88c
configure using validatorsPerNode1/2 instead of chi1/2 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 20:00:52 +01:00
Csaba Kiraly ef4e32ed53
introduce node classes 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 19:57:43 +01:00
Csaba Kiraly dc7a4d3c03
generate row/column interest locally in validator 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 19:54:16 +01:00
Csaba Kiraly 99e051e6fc
adding uplinkBw configuration 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-16 19:50:48 +01:00
Leonardo Bautista-Gomez b4348b0005 Cosmetic changes for documentation 2023-03-03 11:47:27 +01:00
Csaba Kiraly daee84b9ea
add more function docustrings 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 23:59:35 +01:00
Csaba Kiraly 66a9d66dc6
moving helper functions to tools.py 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 22:34:16 +01:00
Csaba Kiraly e611b5143c
refactor dumbRandomScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 11:11:39 +01:00
Csaba Kiraly 3095e440c6
factorize segmentShuffleScheduler code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 11:11:39 +01:00
Csaba Kiraly 2bf85c41a2
factorize send code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 09:54:22 +01:00
Csaba Kiraly b5368b4e43
factorize restore 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 08:41:37 +01:00
Csaba Kiraly 82ee2b5189
simplify dumbRandomScheduler code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 08:38:08 +01:00
Csaba Kiraly f95a393068
improve perNeighborSendQueue 
- improve shuffling between rows and columns
- speed up code execution

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 12:10:45 +01:00
Csaba Kiraly d9a2d5d606
fixup: ensure bw limit is respected 
Lost meaning of return while factorizing schedulers. Fix it
by checking limits after each call.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 12:04:07 +01:00
Csaba Kiraly 0a418b35b2
parametrize dumbRandomScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 10:43:45 +01:00
Csaba Kiraly 300bc19c67
factorize send code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:55:34 +01:00
Csaba Kiraly fa1818a43b
simplify code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:43:18 +01:00
Csaba Kiraly ead127e73e
change defaults to queue per p2p link 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:42:35 +01:00
Csaba Kiraly 89a6b1cdf7
remove old scheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 00:35:17 +01:00
Csaba Kiraly dfacd6bb18
allow push from non-neighbor 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:41 +01:00
Csaba Kiraly 186d430ad1
consider shuffleLines in segmentShuffleScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:40 +01:00
Csaba Kiraly a1a8a4282d
fix scheduler to check result of endSegmentToNeigh 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly c0650bf75a
implement partial line sending logic 
On any given p2p link, it only makes sense to send up to
k messages, after that repair kicks in.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly e70740f530
handle duplicates in receiveRow/Column 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly f91f3da5d2
fixup: segmentShuffleScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly bb55abe2b0
comments only 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 54d101e5b8
segmentShuffleScheduler: persist scheduler state between timesteps 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly cd2b69149b
add segmentShuffleScheduler 
This scheduler check which owned segments needs sending (at least
one neighbor needing it). Then it sends each segment that's worth sending
once, in shuffled order. This is repeated until bw limit.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 655f3a6642
fix: avoid looking into the future 
Checking neigh.receiving is cheating in the current model.
If the timeslot is small, information can't propagate that fast.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly af72e58d08
collect receivedQueue only if it is used later 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 1669ec9236
more debug logging 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:49 +01:00
Csaba Kiraly 5383c59f6f
add shuffleLines and shuffleNeighbors params 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:25:15 +01:00
Csaba Kiraly d0641e4568
add repairOnTheFly parameter 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:25:09 +01:00
Csaba Kiraly f05c3cd233
fix queuing: should queue after repair 
If operation is based on send queues, segments should
be queued after successful repair.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:21:56 +01:00
Csaba Kiraly dff0e5523a
factorize addToSendQueue 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:45 +01:00
Csaba Kiraly 0c91eff67b
add dumbRandomScheduler parameter 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:45 +01:00
Csaba Kiraly b7dab5bad9
fix sendSegmentToNeigh: specify dimension 
Specify along which dimension (row/column) a segment was
sent.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:39 +01:00
Csaba Kiraly 23e40693f1
add perNeighborQueue option 
If enabled, queue incoming messages to outgoing connections on arrival,
as typical in some GossipSub implementations.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly 7c0fcaba78
add validator.perNodeQueue conf option 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly f67c70896c
add to receivedQueue also in row/column receive code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly 9ab51278c8
add shuffledDict helper 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00