codex-storage
/
das-research
mirror of https://github.com/codex-storage/das-research.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
117 Commits 79 Branches 0 Tags 2.6 MiB
Commit Graph

117 Commits

Author SHA1 Message Date
Csaba Kiraly daee84b9ea
add more function docustrings 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 23:59:35 +01:00
Csaba Kiraly 66a9d66dc6
moving helper functions to tools.py 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 22:34:16 +01:00
Csaba Kiraly 68fdaf3572
add method descriptions 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 22:21:31 +01:00
Csaba Kiraly e611b5143c
refactor dumbRandomScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 11:11:39 +01:00
Csaba Kiraly 3095e440c6
factorize segmentShuffleScheduler code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 11:11:39 +01:00
Csaba Kiraly 2bf85c41a2
factorize send code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 09:54:22 +01:00
Csaba Kiraly b5368b4e43
factorize restore 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 08:41:37 +01:00
Csaba Kiraly 82ee2b5189
simplify dumbRandomScheduler code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 08:38:08 +01:00
Csaba Kiraly f95a393068
improve perNeighborSendQueue 
- improve shuffling between rows and columns
- speed up code execution

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 12:10:45 +01:00
Csaba Kiraly d9a2d5d606
fixup: ensure bw limit is respected 
Lost meaning of return while factorizing schedulers. Fix it
by checking limits after each call.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 12:04:07 +01:00
Csaba Kiraly 0a418b35b2
parametrize dumbRandomScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 10:43:45 +01:00
Csaba Kiraly a03371cf4e
add logging of TX and RX statistics 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 10:43:45 +01:00
Csaba Kiraly 300bc19c67
factorize send code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:55:34 +01:00
Csaba Kiraly fa1818a43b
simplify code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:43:18 +01:00
Csaba Kiraly ead127e73e
change defaults to queue per p2p link 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:42:35 +01:00
Csaba Kiraly 89a6b1cdf7
remove old scheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 00:35:17 +01:00
Csaba Kiraly 2707269836
fixup: moving simulator config to resetShape 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 22:21:16 +01:00
Csaba Kiraly dfacd6bb18
allow push from non-neighbor 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:41 +01:00
Csaba Kiraly b33f829b0e
proposer might push segments without participating in mesh 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:41 +01:00
Csaba Kiraly 186d430ad1
consider shuffleLines in segmentShuffleScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:40 +01:00
Csaba Kiraly a1a8a4282d
fix scheduler to check result of endSegmentToNeigh 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly c0650bf75a
implement partial line sending logic 
On any given p2p link, it only makes sense to send up to
k messages, after that repair kicks in.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly e70740f530
handle duplicates in receiveRow/Column 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly f91f3da5d2
fixup: segmentShuffleScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly bb55abe2b0
comments only 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 54d101e5b8
segmentShuffleScheduler: persist scheduler state between timesteps 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly cd2b69149b
add segmentShuffleScheduler 
This scheduler check which owned segments needs sending (at least
one neighbor needing it). Then it sends each segment that's worth sending
once, in shuffled order. This is repeated until bw limit.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 655f3a6642
fix: avoid looking into the future 
Checking neigh.receiving is cheating in the current model.
If the timeslot is small, information can't propagate that fast.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly af72e58d08
collect receivedQueue only if it is used later 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 1669ec9236
more debug logging 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:49 +01:00
Csaba Kiraly 5383c59f6f
add shuffleLines and shuffleNeighbors params 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:25:15 +01:00
Csaba Kiraly d0641e4568
add repairOnTheFly parameter 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:25:09 +01:00
Csaba Kiraly f05c3cd233
fix queuing: should queue after repair 
If operation is based on send queues, segments should
be queued after successful repair.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:21:56 +01:00
Csaba Kiraly dff0e5523a
factorize addToSendQueue 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:45 +01:00
Csaba Kiraly 0c91eff67b
add dumbRandomScheduler parameter 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:45 +01:00
Csaba Kiraly b7dab5bad9
fix sendSegmentToNeigh: specify dimension 
Specify along which dimension (row/column) a segment was
sent.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:39 +01:00
Csaba Kiraly 23e40693f1
add perNeighborQueue option 
If enabled, queue incoming messages to outgoing connections on arrival,
as typical in some GossipSub implementations.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly 7c0fcaba78
add validator.perNodeQueue conf option 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly f67c70896c
add to receivedQueue also in row/column receive code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly 9ab51278c8
add shuffledDict helper 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly 1403ca7ad0
add random scheduler 2023-02-23 21:13:39 +01:00
Csaba Kiraly 0f4883bf26
add node level send queue 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:35 +01:00
Csaba Kiraly 382954de02
add segment level send/receive 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:09:59 +01:00
Csaba Kiraly 3fc7455c0b
reduce default BW to more interesting values 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:09:59 +01:00
Csaba Kiraly 3917001e6a
send one segment at a time 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:09:59 +01:00
Csaba Kiraly eb277d9b43
limit batchsize of sending from a line 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:09:59 +01:00
Csaba Kiraly 07437ddde8
fixup bwUplink check (still approximate) 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:08:26 +01:00
Csaba Kiraly bb8d05257b
WIP: initial implementation of uplink bandwidth limit 
- approximate: BW is not handled strict, entire rows are sent and can go over limit
- WIP: work in progress implementation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:08:17 +01:00
Leo b7877f6130
Merge pull request #16 from status-im/more-config 
Add some more configuration parameters and some more testing
 2023-02-23 14:36:59 +01:00
Leonardo Bautista-Gomez 6e42055cb9 Add some more configuration parameters and some more testing 2023-02-23 12:17:57 +01:00