* Refactor RequestBroker to support context aware use - introduction of BrokerContext
* Context aware extension for EventBroker, EventBoker support for native or external types
* Enhance MultiRequestBroker - similar to RequestBroker and EventBroker - with support for native and external types and context aware execution.
* Move duplicated and common code into broker_utils from event- request- and multi_request_brokers
* Change BrokerContext from random number to counter
* Apply suggestion from @Ivansete-status
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Adjust naming in broker tests
* Follow up adjustment from send_api use
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Adapt using chronos' TokenBucket. Removed TokenBucket and test. bump nim-chronos -> nim-libp2p/nim-lsquic/nim-jwt -> adapt to latest libp2p changes
* Fix libp2p/utility reports unlisted exception can occure from close of socket in waitForService - -d:ssl compile flag caused it
* Adapt request_limiter to new chronos' TokenBucket replenish algorithm to keep original intent of use
* Fix filter dos protection test
* Fix peer manager tests due change caused by new libp2p
* Adjust store test rate limit to eliminate CI test flakyness of timing
* Adjust store test rate limit to eliminate CI test flakyness of timing - lightpush/legacy_lightpush/filter
* Rework filter dos protection test to avoid CI crazy timing causing flakyness in test results compared to local runs
* Rework lightpush dos protection test to avoid CI crazy timing causing flakyness in test results compared to local runs
* Rework lightpush and legacy lightpush rate limit tests to eliminate timing effect in CI that cause longer awaits thus result in minting new tokens unlike local runs
* chore: extend RequestBroker with supporting native and external types and added possibility to define non-async (aka sync) requests for simplicity and performance
* Adapt gcsafe pragma for RequestBroker sync requests and provider signatures as requirement
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Introduce EventBroker and RequestBroker as decoupling helpers that represent reactive (event-driven) and proactive (request/response) patterns without tight coupling between modules
* Address copilot observation. error log if failed listener call exception, handling listener overuse - run out of IDs
* Address review observations: no exception to leak, listeners must raise no exception, adding listener now reports error with Result.
* Added MultiRequestBroker utility to collect results from many providers
* Support an arbitrary number of arguments for RequestBroker's request/provider signature
* MultiRequestBroker allows provider procs to throw exceptions, which will be handled during request processing.
* MultiRequestBroker supports one zero arg signature and/or multi arg signature
* test no exception leaks from RequestBroker and MultiRequestBroker
* Embed MultiRequestBroker tests into common
* EventBroker: removed all ...Broker typed public procs to simplify EventBroker interface, forger is renamed to dropListener
* Make Request's broker type private
* MultiRequestBroker: Use explicit returns in generated procs
* Updated descriptions of EventBroker and RequestBroker, updated RequestBroker.setProvider, returns error if already set.
* Better description for MultiRequestBroker and its usage
* Add EventBroker support for ref objects, fix emit variant with event object ctor
* Add RequestBroker support for ref objects
* Add MultiRequestBroker support for ref objects
* Mover brokers under waku/common
* update rendezvous to work with WakuPeeRecord and use libp2p updated version
* split rendezvous client and service implementation
* mount rendezvous client by default
* introduce createNode
# Conflicts:
# apps/wakunode2/cli_args.nim
* remove confutils dependency on the library
* test: remove websocket in default test config
* update to latest specs
* test: cli_args
* align to spec changes (sovereign, message conf, entrypoints
* accept enr, entree and multiaddr as entry points
* post rebase
* format
* change from "sovereign" to "core"
* add example
* get example to continue running
* nitpicks
* idiomatic constructors
* fix enum naming
* replace procs with consts
* remove messageConfirmation
* use pure enum
* rename example file
Split `WakuNodeConfig` object for better separation of concerns and to introduce a tree-like structure to configuration.
* fix: ensure twn cluster conf is still applied when clusterId=1
* test: remove usage of `WakuNodeConf`
* Remove macro, split builder files, remove wakunodeconf from tests
* rm network_conf_builder module as it is not used
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* Enhance metrics labels
* Bound the metrics-label-values in arbitrary queries
* The metrics-label-values for prepared statements are kept as
they already represent a fixed set
* undo apt install libpcre (not circuit-relay related.)
* nat.nim: protect against possible exceptions when calling getExternalIP
* new external CLI argument, isRelayClient
* waku factory change to mount circuit hop proto by default
* waku_node: move autonat_service to a separate module
* Enhanced peer-ex protocol - added rate limiting, added response status and desc to the rpc
* Better error result handling for PeerEx request, adjusted tests
* Refactored RateLimit configuration option for better CLI UX - now possible to set separate limits per protocol. Adjusted mountings. Added and adjusted tests
* Fix libwaku due to changes of error return type of fetchPeerExchangePeers
* Fix rate limit setting tests due to changed defaults
* Introduce new gauge to help dasboard effectively show current rate limit applied for protocol
* Adjust timeing in filter rate limit test to let macos CI test run ok.
* Address review findings, namings, error logs, removed left-overs
* Changes to reflect latest spec agreement and changes. PeerExchange RPC is changed the now respond structure will contain status_code and status_desc.
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
(although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
* Added proper per shard bandwidth metric calculation and proper logging of in/out messages
Changed rate limit metrics for dashboard
Updated monitoring dashboard for bw and rate metrics
* Prevent multiple nat module initialization that cause dead lock in nat refresh thread tear down during tests.
* NPROC to 1 to avoid parallel test runs can lead to timing and port allocation issues
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
