codex-storage
/
das-research
mirror of https://github.com/codex-storage/das-research.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
238 Commits 79 Branches 0 Tags 2.6 MiB
Commit Graph

238 Commits

Author SHA1 Message Date
Csaba Kiraly 16b670e916
fix issues with logging in parallel execution 
For fixing logging issues see
https://stackoverflow.com/questions/58026381/logging-nested-functions-using-joblib-parallel-and-delayed-calls
and https://github.com/joblib/joblib/issues/1017

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-07 13:11:17 +01:00
Csaba Kiraly f5ffb0a07b
use joblib to run in parallel 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-07 13:11:13 +01:00
Csaba Kiraly 0b6cfad967
factorize study code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-07 13:10:33 +01:00
Csaba Kiraly 0a5afd97de
remove old config files 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-07 12:10:36 +01:00
Csaba Kiraly 84e5482a6d
improve config filename error handling 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-07 12:07:31 +01:00
Csaba Kiraly 4a4f02427c
change config language to simple (or complex) code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-07 12:06:57 +01:00
Leo 37ff89bd82
Merge pull request #17 from status-im/bandwidth 
Bandwidth limited diffusion
 2023-03-03 11:48:21 +01:00
Leonardo Bautista-Gomez b4348b0005 Cosmetic changes for documentation 2023-03-03 11:47:27 +01:00
Csaba Kiraly 35d1790429
Merge remote-tracking branch 'origin/vis' into develop 
# Conflicts:
#	.gitignore
#	DAS/simulator.py
#	study.py
 2023-03-02 01:32:25 +01:00
Csaba Kiraly daee84b9ea
add more function docustrings 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 23:59:35 +01:00
Csaba Kiraly 66a9d66dc6
moving helper functions to tools.py 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 22:34:16 +01:00
Csaba Kiraly 68fdaf3572
add method descriptions 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 22:21:31 +01:00
Csaba Kiraly e611b5143c
refactor dumbRandomScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 11:11:39 +01:00
Csaba Kiraly 3095e440c6
factorize segmentShuffleScheduler code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 11:11:39 +01:00
Csaba Kiraly 2bf85c41a2
factorize send code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 09:54:22 +01:00
Csaba Kiraly b5368b4e43
factorize restore 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 08:41:37 +01:00
Csaba Kiraly 82ee2b5189
simplify dumbRandomScheduler code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-03-01 08:38:08 +01:00
HajarZaiz 57f3509102 Removed Frontend directory from Git index 2023-02-26 18:38:20 +01:00
HajarZaiz 39c454d3f4 minor plot fixes 2023-02-26 18:36:02 +01:00
HajarZaiz ceb8357034 Merge branch 'vis' of https://github.com/status-im/das-research into vis 2023-02-26 18:27:47 +01:00
Csaba Kiraly f95a393068
improve perNeighborSendQueue 
- improve shuffling between rows and columns
- speed up code execution

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 12:10:45 +01:00
Csaba Kiraly d9a2d5d606
fixup: ensure bw limit is respected 
Lost meaning of return while factorizing schedulers. Fix it
by checking limits after each call.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 12:04:07 +01:00
Csaba Kiraly 0a418b35b2
parametrize dumbRandomScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 10:43:45 +01:00
Csaba Kiraly a03371cf4e
add logging of TX and RX statistics 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 10:43:45 +01:00
Csaba Kiraly 300bc19c67
factorize send code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:55:34 +01:00
Csaba Kiraly fa1818a43b
simplify code 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:43:18 +01:00
Csaba Kiraly ead127e73e
change defaults to queue per p2p link 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 08:42:35 +01:00
Csaba Kiraly 89a6b1cdf7
remove old scheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-24 00:35:17 +01:00
Csaba Kiraly 2707269836
fixup: moving simulator config to resetShape 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 22:21:16 +01:00
Csaba Kiraly dfacd6bb18
allow push from non-neighbor 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:41 +01:00
Csaba Kiraly b33f829b0e
proposer might push segments without participating in mesh 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:41 +01:00
Csaba Kiraly 186d430ad1
consider shuffleLines in segmentShuffleScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:28:40 +01:00
Csaba Kiraly a1a8a4282d
fix scheduler to check result of endSegmentToNeigh 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly c0650bf75a
implement partial line sending logic 
On any given p2p link, it only makes sense to send up to
k messages, after that repair kicks in.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly e70740f530
handle duplicates in receiveRow/Column 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly f91f3da5d2
fixup: segmentShuffleScheduler 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:53 +01:00
Csaba Kiraly bb55abe2b0
comments only 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 54d101e5b8
segmentShuffleScheduler: persist scheduler state between timesteps 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly cd2b69149b
add segmentShuffleScheduler 
This scheduler check which owned segments needs sending (at least
one neighbor needing it). Then it sends each segment that's worth sending
once, in shuffled order. This is repeated until bw limit.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 655f3a6642
fix: avoid looking into the future 
Checking neigh.receiving is cheating in the current model.
If the timeslot is small, information can't propagate that fast.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly af72e58d08
collect receivedQueue only if it is used later 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:52 +01:00
Csaba Kiraly 1669ec9236
more debug logging 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:27:49 +01:00
Csaba Kiraly 5383c59f6f
add shuffleLines and shuffleNeighbors params 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:25:15 +01:00
Csaba Kiraly d0641e4568
add repairOnTheFly parameter 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:25:09 +01:00
Csaba Kiraly f05c3cd233
fix queuing: should queue after repair 
If operation is based on send queues, segments should
be queued after successful repair.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:21:56 +01:00
Csaba Kiraly dff0e5523a
factorize addToSendQueue 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:45 +01:00
Csaba Kiraly 0c91eff67b
add dumbRandomScheduler parameter 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:45 +01:00
Csaba Kiraly b7dab5bad9
fix sendSegmentToNeigh: specify dimension 
Specify along which dimension (row/column) a segment was
sent.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:15:39 +01:00
Csaba Kiraly 23e40693f1
add perNeighborQueue option 
If enabled, queue incoming messages to outgoing connections on arrival,
as typical in some GossipSub implementations.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00
Csaba Kiraly 7c0fcaba78
add validator.perNodeQueue conf option 
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2023-02-23 21:13:39 +01:00