* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
(although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
* Added proper per shard bandwidth metric calculation and proper logging of in/out messages
Changed rate limit metrics for dashboard
Updated monitoring dashboard for bw and rate metrics
* Prevent multiple nat module initialization that cause dead lock in nat refresh thread tear down during tests.
* NPROC to 1 to avoid parallel test runs can lead to timing and port allocation issues
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
* feat: Added simple, configurable rate limit for lightpush and store-query
Adjust lightpush rest response to rate limit, added tests ann some fixes
Add rest store query test for rate limit checks and proper error response
Update apps/wakunode2/external_config.nim
Move chronos/tokenbucket to nwaku codebasee with limited and fixed feature set
Add meterics counter to lightpush rate limits
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
* message.nim: set max message size to 150KiB according to spec
Using KiB instead of KB because that seems more aligned with
the actual default defined in nim-libp2p (1024 * 1024)
Spec details: https://rfc.vac.dev/spec/64/#message-size
* test_protocol.nim: align test to current WakuMessage limit
* test_waku_client.nim: adapt test to MaxWakuMessageSize change
* make maxMessageSize configurable for wakunode2
* wakunode2 app now accepts max-num-bytes-msg-size with KiB, KB, or B units
* testlib/wakunode.nim: set maxMessageSize: "1024 KiB"
* test_waku_client.nim: remove duplicate check in "Valid Payload Sizes"
* set DefaultMaxWakuMessageSizeStr as the only source of truth
* external_config.nim: rename max-num-bytes-msg-size -> max-msg-size
* on_chain/group_manager: use .async: (raises:[Exception]).
* bump nim-dnsdisc
* update nim-chronos to the latest state
* chat2.nim: catch any possible exception when stopping
* chat2bridge.nim: make it to compile after vendor bump
* ValidIpAddress (deprecated) -> IpAddress
* vendor/nim-libp2p additional bump
* libwaku: adapt to vendor bump
* testlib/wakunode.nim: adapt to vendor bump (ValidIpAddress -> IpAddress)
* waku_node: avoid throwing any exception from stop*(node: WakuNode)
* test_confutils_envvar.nim: ValidIpAddress -> IpAddress
* test_jsonrpc_store: capture exception
* test_rln*: handling exceptions
* adaptation to make test_rln_* to work properly
* signature enhancement of group_manager methods
* db_postgres: use prepared statements on most freq select queries
* db_postgres/dbconn.nim adding better feedback in case of query error
* dbconn: use of isOkOr
* pgasyncpool: refactor to reduce code (valueOr, catch:)
* db_postgres, postgres_driver: better performance by using callback.
There were a bunch of milliseconds being lost due to multiple-row
processing. This commit aims to have the minimum possible row
process time.
* pgasyncpool: clarifying logic around pool conn management.
* db_postgres: removing duplicate code and more searchable proc names.
* chore: add retention policy with GB or MB limitation #1885
* chore: add retention policy with GB or MB limitation
* chore: updated code post review- retention policy
* ci: extract discordNotify to separate file
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* ci: push images to new wakuorg/nwaku repo
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* ci: enforce default Docker image tags strictly
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* ci: push GIT_REF if it looks like a version
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* fix: update wakuv2 fleet DNS discovery enrtree
https://github.com/status-im/infra-misc/issues/171
* chore: resolving DNS IP and publishing it when no extIp is provided (#2030)
* feat(coverage): Add simple coverage (#2067)
* Add test aggregator to all directories.
* Implement coverage script.
* fix(ci): fix name of discord notify method
Also use absolute path to load Groovy script.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version (#2080)
* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version
* add more metrics, refactor how most metrics are calculated
* rework metrics table fillup
* reset connErr to make sure we honour successful reconnection
* chore(cbindings): Adding cpp example that integrates the 'libwaku' (#2079)
* Adding cpp example that integrates the `libwaku`
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* fix(ci): update the dependency list in pre-release WF (#2088)
* chore: adding NetConfig test suite (#2091)
---------
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Anton Iakimov <yakimant@gmail.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>
Co-authored-by: Vaclav Pavlin <vaclav@status.im>
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* dburl.nim: simpler regex that can support db_urls with . and - in hostname
* dbrul.nim: accepting any non-empty sequence for user and password
* dburl.nim: skipping validation for 'sqlite' db paths
It starts an asynchronous infinite task that checks the connectivity
with the database. In case of error, the postgres_healthcheck task
tries to reconnect for a while, and if it determines that the connection
cannot be resumed, then it invokes a callback indicating that
situation. For the case of the `wakunode2` app, this callback
quits the application itself and adds a log trace indicating
the connectivity issue with the database.
* Refactoring the Waku Archive. Simplifying the app.nim
This change is needed to accommodate the further PRs where we will integrate Postgres in `wakunode2`.
Notice that I had to adapt to use 'rlpx_connected_peers' instead
of 'connected_peers' in 'wakunode1.nim' because due to the update
of the 'vendor/nim-eth', which adds the dependency-break with
'confutils' but also includes another changes.
Aside note: we cannot have 'confutils' dependency in 'nim-eth' because
that will prevent the generation of any waku dynamic library.
* feat(cbindings): first commit - waku relay (#1632)
* test_app.nim: fix compilation issue. App.init(..) -> App.new(..)
* Simplifying library name (libwaku) and standardizing function names (waku_*)
* Proper wrapper of the waku_node API and creation of the libwaku.a
* Rolling back changes that are not needed
* Rolling back changes that are out of the scope of this task
* wakunode.nim: Removing unnecessary import
* Aplying PR suggestions
* Renaming 'waku.h' -> 'libwaku.h'
* Use of 'isNil' instead of '== nil'
* libwaku.nim: explicitly setting waku_poll() as gcsafe
