* Enhanced peer-ex protocol - added rate limiting, added response status and desc to the rpc
* Better error result handling for PeerEx request, adjusted tests
* Refactored RateLimit configuration option for better CLI UX - now possible to set separate limits per protocol. Adjusted mountings. Added and adjusted tests
* Fix libwaku due to changes of error return type of fetchPeerExchangePeers
* Fix rate limit setting tests due to changed defaults
* Introduce new gauge to help dasboard effectively show current rate limit applied for protocol
* Adjust timeing in filter rate limit test to let macos CI test run ok.
* Address review findings, namings, error logs, removed left-overs
* Changes to reflect latest spec agreement and changes. PeerExchange RPC is changed the now respond structure will contain status_code and status_desc.
* Switch libnegentropy library build from shared to static linkage
* Update negentropy with -fPIC compile option that is necessary for libwaku build
* Bump waku-org/negentropy to the latest on master to incorporate merged static build of libnegentropy
- Use of observer observable pattern to inform delivery_monitor about subscription state
- send_monitor becomes a publish observer of lightpush and relay
- deliver monitor add more protection against possible crash and better logs
- creating a separate proc in store client for delivery monitor
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
(although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
* chore(rln): rln message limit to 100 (#2883)
* postgres_driver: add more error handling when creating partitions
Given that multiple nodes can be connected to the same database,
it can happen that other node did something that my node was willing
to do. In this commit, we overcome the possible "interleaved"
partition creation.
---------
Co-authored-by: Alvaro Revuelta <alvrevuelta@gmail.com>
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
The id column is needed because it contains the message digest
which is used in store v2, and we need to keep support to
store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
We want the nwaku node to simultaneously support store-v2
requests and store-v3 requests.
Only the legacy archive is in charge of archiving messages, and the
archived information is suitable to fulfill both store-v2 and
store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
* chore: add unit test for testing duplicate message push with timedcache
* chore: update according to better naming convention
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* Added proper per shard bandwidth metric calculation and proper logging of in/out messages
Changed rate limit metrics for dashboard
Updated monitoring dashboard for bw and rate metrics
* fix: it's resolve duplicate message forwarding for filter service
* chore: update little flow
* fix: update implementation using timed cache method
* chore: simple format change
* chore: simple format change
* chore: update put function location
* chore: update according suggestion
* chore: update logs with topic discription & debug msg
* chore: update unit according to error msg
* chore: update rest unit according to error msg
* chore: add content-topic with debug msg
With the original approach it happened cases where one connection
acquired the lock and other connection tried to release the same lock,
causing an unrecoverable failure which made the node to end.
* Prevent multiple nat module initialization that cause dead lock in nat refresh thread tear down during tests.
* NPROC to 1 to avoid parallel test runs can lead to timing and port allocation issues
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
* archive.nim: reduce the database report interval from 1 to 30 min
This counts the number of rows with "select count(1) from messages"
which is quite intense and we shouldn't run it every minute
* aside cleanup
NagyZoltanPeter
gabrielmer
Ivan FB
fryorcraken
Simon-Pierre Vivier
Darshan K
richΛrd
Aaryamann Challani
Prem Chaitanya Prathi
Sasha
Alvaro Revuelta
Ivan Folgueira Bande
Akhil
Darshan K