Commit Graph

100 Commits

Author SHA1 Message Date
Alberto Soutullo 5b852d5817
Added lazy batch for nimlibp2p 2024-06-13 17:20:08 +02:00
Alberto Soutullo aadde501d9
Change script to work in lab 2024-06-03 18:37:34 +02:00
Alberto Soutullo 1cd5e7106c
Add files for batch simulation 2024-06-03 18:32:51 +02:00
Alberto Soutullo 6786960c92
Nim-libp2p deploy (#31)
* Add yamls for nim-libp2p deployment

* Delete old deployment file

* Rename folder
2024-05-29 11:01:58 +02:00
Alberto Soutullo 7c0a786536
Publisher changes (#30)
* Add:
- Nº of message
- Handle exceptions
- Correctly sent KBytes
- Send X nº of messages

* Change default value for delay
2024-05-27 19:52:45 +01:00
Alberto Soutullo 07ef73dd58
Update yamls (#29) 2024-05-24 13:59:15 +02:00
Alberto Soutullo 249f35a970
Add new parameter for selecting how many datapoints we will add to the plots (#28) 2024-05-24 11:56:38 +02:00
Alberto Soutullo a6a21a4045
Extend query to multiple timestamps (#27)
* Added midstrap for ordering

* Changed scrapper for query a batch of timestamps

* Changed debug level to better tracking

* Updated scrape to have as example
2024-05-23 16:22:25 +02:00
Alberto Soutullo 2a32cb468a
Improved logging and error checking (#26) 2024-05-23 15:50:24 +02:00
Alberto Soutullo 86445bab0c
Alberto/graph visualization into main (#25)
* WIP: Create plot using qt5 in jupyter. Select from first timestamp to given one, and message hash.

* WIP: First approach efficiently replot subgraph saving positions

* WIP: Maintain ax limits between plots for easier visualization

* WIP: Plot elapsed time on edges with msg hash

* WIP: Correctly calculate elapsed time based on msg hash.

* WIP: Change multiindex order. Now dropdown only shows timestamps related to msg hash

* WIP: Moved functions to external files. Figure crashes with weird behavior.

* WIP: Fix Python crashing with jupyter backend
2024-05-22 13:02:24 +02:00
Alberto Soutullo c5ce9dbf43
Msg tracker - parsing (#24)
* WIP: Parsing and creating df from log files done

* First tracking version structure

* Add PR suggestions
2024-05-13 16:42:24 +02:00
Alberto Soutullo 8fe63202b8
Download pod logs (#21)
* Added method to download pod logs

* Fixed typo in ok_value

* Added kubeconfig in yaml file

* Added missing comment
2024-04-15 16:01:00 +02:00
Alberto Soutullo 1be49b98e1
Update tests scrapper and utils (#20)
* Updated tests in scrapper class

* Added utils tests

* Deleted old files

* Added PR suggestions.
2024-04-09 10:02:41 +02:00
Alberto Soutullo a50b182126
Update plotter and data classes tests (#19)
* Delete unnecessary method for concat dfs.

* Added tests for data_handler

* Deleted todo as it is not needed

* Added tests for data file handler

* Moved add file as mean to parent class

* Fixed new classes for more than one folder usecase

* Adjusted test data handler to be more understandable

* Fixed tests in data file handler

* Moved test to correspondant classes

* Renamed variables

* Minor format

* Added tests for data request handler

* Added tests for plotter

* Removed unnecessary comments

* Changed setUp for setUpClass

* Changed test data files to resources folder
2024-04-03 18:12:32 +02:00
Alberto Soutullo 2ca551ce44
Yaml improvements - plotter (#18)
* Add new fields to yaml

* Using new yaml parameters in plotter class

* Update src/plotting/plotter.py

string format

Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>

---------

Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>
2024-03-27 18:04:34 +01:00
Alberto Soutullo 657562b198
Scrape yaml improvements (#17)
* Changed YAML to possible variance

* Remove out folder from Scrapper parameters

* Corrected YAML indentation

* Use correct yaml format to create_query

* Use correct yaml format to when using dump data from scrapper

* Removed file utils and correctly move them to path

* Final yaml version

* Removed unnecessary list from yaml structure

* Use Path.parent, and remove wrongly added methods

* Removed other wrongly added function hehe
2024-03-27 16:41:17 +01:00
Alberto Soutullo 52cc8b4396
Alberto/data manager (#16)
* WIP: Trying portforward

* Created Kubernetes class to manage api inside scraper class.

* Modify scrapper class to use kubernetes class

* Extracted get query data to utils and deleted unused function

* Cleaned main.py

* Fix bug with pod name

* Typing and cleaning

* Extract order function to list utils file

* Extract path function to path utils file

* Created data handler class

* Changed scrapper to use data handler class

* Cleaned imports

* Alex PR suggestions

* Removed unnecessary if

* Extract order function to list utils file

* Extract path function to path utils file

* Created data handler class

* Changed scrapper to use data handler class

* Cleaned imports

* Add function to file_utils.py

* Created structure of data handler classes

* Created Parent class for data handler

* Modified data file handler

* Created data request handler

* Changed file utils to use Path instead of str

* Use correct class in scrapper

* Abstracted plotter to use data handler

* Added some typing left

* Modify scrapper class to use kubernetes class

* Extract order function to list utils file

* Extract path function to path utils file

* Created data handler class

* Changed scrapper to use data handler class

* Cleaned imports

* Changed scrapper to use data handler class

* Cleaned imports

* Add function to file_utils.py

* Created structure of data handler classes

* Created Parent class for data handler

* Modified data file handler

* Created data request handler

* Changed file utils to use Path instead of str

* Use correct class in scrapper

* Abstracted plotter to use data handler

* Added some typing left

* Remove redundant method for check if a Path exists

* Changed Ok content return from message to object.

* Changed get_dataframe method for property, and moved it to parent class

* Removed unnecessary method to get the file name of a Path

* Changed list comprehension to use Path instead of os
2024-03-27 15:16:55 +01:00
Alberto Soutullo 774d544b06
Add Thanos portforward (#15)
* WIP: Trying portforward

* Created Kubernetes class to manage api inside scraper class.

* Modify scrapper class to use kubernetes class

* Extracted get query data to utils and deleted unused function

* Cleaned main.py

* Fix bug with pod name

* Typing and cleaning

* Alex PR suggestions

* Removed unnecessary if
2024-03-21 13:08:27 +01:00
Alberto Soutullo 2b60bf6555
Merge pull request #13 from vacp2p/plotter
Plotter into master
2024-03-14 15:21:15 +01:00
Alberto Soutullo 704c38eaf2
Merge remote-tracking branch 'origin/plotter' into plotter
# Conflicts:
#	scrape.yaml
#	src/utils/file_utils.py
2024-03-14 15:19:57 +01:00
Alberto Soutullo ff22e16d3f
Added readme for plotter. 2024-03-14 15:17:55 +01:00
Alberto Soutullo 103d3b724d
WIP: test for plotter 2024-03-14 15:17:54 +01:00
Alberto Soutullo f14c66bf3c
Refactored plotter 2024-03-14 15:17:53 +01:00
Alberto Soutullo 9afc94b28a
Fixed logger to avoid logging external libraries 2024-03-14 15:17:52 +01:00
Alberto Soutullo 42fc400c8d
Fixed scrapper test 2024-03-14 15:17:51 +01:00
Alberto Soutullo 6c0f119498
Changed Unix Timestamp for Time when creating csv 2024-03-14 15:17:51 +01:00
Alberto Soutullo 0fc2fb967e
Improved plotter.py 2024-03-14 15:17:49 +01:00
Alberto Soutullo f1a4f65459
Cleaned plotter and modified yaml file. Also added np to requirements 2024-03-14 15:17:19 +01:00
Alberto Soutullo 5c6e928516
First Logger wip version 2024-03-14 15:17:16 +01:00
Alberto Soutullo 569f144415
WIP: Added versions of plotters 2024-03-14 15:14:23 +01:00
Alberto Soutullo 0a7d508366
Merge pull request #12 from vacp2p/metrics_scrapper
Improve scrape interval
2024-03-14 15:09:26 +01:00
Alberto Soutullo b8353f3624
Fixed tests due previous changes 2024-03-14 15:01:41 +01:00
Alberto Soutullo d75eb1397e
Changed scrapper to select initial and final time 2024-03-14 15:00:36 +01:00
Alberto Soutullo 933e231e14
Added readme for plotter. 2024-03-14 13:05:06 +01:00
Alberto Soutullo 1cda7a9c11
WIP: test for plotter 2024-03-14 13:04:59 +01:00
Alberto Soutullo bed4592fd3
Refactored plotter 2024-03-14 13:04:26 +01:00
Alberto Soutullo 098d6debaf
Fixed logger to avoid logging external libraries 2024-03-13 18:31:18 +01:00
Alberto Soutullo cba16436e6
Merge pull request #5 from vacp2p/metrics_scrapper
Metrics scrapper to main
2024-03-07 11:39:39 +01:00
Alberto Soutullo 23e22d8073
Fixed scrapper test 2024-03-07 11:37:33 +01:00
Alberto Soutullo 4cb8ed0911
Changed Unix Timestamp for Time when creating csv 2024-03-07 11:34:27 +01:00
Alberto Soutullo 52174bcfd0
Improved plotter.py 2024-03-07 11:33:56 +01:00
Benjamin Arntzen 0f133d523d Also cleanup publisher 2024-03-06 15:24:51 +00:00
Benjamin Arntzen 8c99c71320 Update midstrap and bootstrap to separate them from nodes 2024-03-06 14:50:31 +00:00
Benjamin Arntzen 66c2fd4da2 Updates for bootstrap-midstrap separation from node service 2024-03-06 13:55:57 +00:00
Benjamin Arntzen ce3a3d878b Fixes for various settings 2024-03-05 20:33:43 +00:00
Benjamin Arntzen 0b94673cc1 Lower to 3 ENRs 2024-03-04 13:51:13 +00:00
Benjamin Arntzen 55048d1f24 Change max connections 2024-03-04 13:48:13 +00:00
Benjamin Arntzen 00855d8a12 Updated deployment script 2024-03-04 13:44:34 +00:00
Benjamin Arntzen 584bf53d0f Add accelerated deployment publisher 2024-03-04 13:43:35 +00:00
Benjamin Arntzen 27b6604fc0 Update accelerated deployment with successful params 2024-03-04 12:52:58 +00:00