891 Commits

Author SHA1 Message Date
fryorcraken
4d5c152f5b
feat: introduce reliable channels (#2526)
* SDS: pushOutgoingMessage is actually sync

* SDS: ensure that `ContentMessage` class is stored in local history with `valueOf` method

* feat: introduce reliable channels

Easy to use Scalable Data Sync (SDS, e2e reliability) wrapper, that includes:
- store queries upon connection to store nodes
- store queries to retrieve missing messages

* remove `channel` prefix

* attempt to improve performance when processing a lot of incoming messages

* test: split test file

* use index.ts for re-export only.

* improve if condition

* use getter for isStarted

* waku node already auto-start

* rename send

* fix lightPush.send type post rebase

* test: remove extra console.log

* SDS: emit messages as missing as soon as they are received

* make configurable elapse time for task process

* typo

* use string instead of enum for event types

* ReliableChannel.send returns the message id
2025-09-09 12:43:48 +10:00
Sasha
8f09f5fa5a
fix: nwaku master interop tests (#2625)
* fix REST endpoint

* disable px in peer cache test
2025-09-08 22:48:18 +02:00
Arseniy Klempner
16253026c6
feat: implement lp-v3 error codes with backwards compatibility (#2501)
* feat: implement LightPush v3 protocol support

Add comprehensive LightPush v3 protocol implementation with:

Core Features:
- LightPush v3 protocol codec and multicodec detection
- Status code-based error handling and validation
- Protocol version inference and compatibility layers
- Enhanced error types with detailed failure information

Protocol Support:
- Automatic v3/v2 protocol negotiation and fallback
- Status code mapping to LightPush error types
- Protocol version tracking in SDK results
- Mixed protocol environment support

Testing Infrastructure:
- Comprehensive v3 error code handling tests
- Mock functions for v3/v2 response scenarios
- Protocol version detection and validation tests
- Backward compatibility verification

Implementation Details:
- Clean separation between v2 and v3 response handling
- Type-safe status code validation with isSuccess helper
- Enhanced failure reporting with protocol version context
- Proper error propagation through SDK layers

This implementation maintains full backward compatibility with v2
while providing enhanced functionality for v3 protocol features.

* feat: handle both light push protocols

* fix: unsubscribe test

* feat: consolidate lpv2/v3 types

* feat(tests): bump nwaku to 0.36.0

* fix: remove extraneous exports

* fix: add delay to tests

* fix: remove protocol result types

* feat: consolidate light push codec branching

* fix: revert nwaku image

* fix: remove multicodec

* fix: remove protocolversion

* feat: simplify v2/v3 branching logic to use two stream managers

* fix: remove unused utils

* fix: remove comments

* fix: revert store test

* fix: cleanup lightpush sdk

* fix: remove unused util

* fix: remove unused exports

* fix: rename file from public to protocol_handler

* fix: use proper type for sdk result

* fix: update return types in filter

* fix: rebase against latest master

* fix: use both lightpush codecs when waiting for peer

* fix: handle both lp codecs

* fix: remove unused code

* feat: use array for multicodec fields

* fix: add timestamp if missing in v3 rpc

* fix: resolve on either lp codec when waiting for peer

* fix: remove unused util

* fix: remove unnecessary abstraction

* feat: accept nwaku docker image as arg, test lp backwards compat

* fix: revert filter error

* feat: add legacy flag to enable lightpushv2 only

* Revert "feat: accept nwaku docker image as arg, test lp backwards compat"

This reverts commit 857e12cbc73305e5c51abd057665bd34708b2737.

* fix: remove unused test

* feat: improve lp3 (#2597)

* improve light push core

* move back to singualar multicodec property, enable array prop only for light push

* implement v2/v3 interop e2e test, re-add useLegacy flag, ensure e2e runs for v2 and v3

* fix v2 v3 condition

* generate message package earlier

* add log, fix condition

---------

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
Co-authored-by: Sasha <oleksandr@status.im>
2025-09-05 00:52:37 +02:00
Sasha
a7f30b1211
feat: add debounce to health indicator (#2594) 2025-09-04 00:49:10 +02:00
Sasha
2fba052b8b
feat: add start/stop to filter (#2592) 2025-08-30 01:32:59 +02:00
fryorcraken
914beb6531
fix: (sds) ensure incoming messages have their retrieval hint stored (#2604)
* SDS: export `MessageId`

* SDS: attach retrieval hints to incoming messages

* sds: ensure items are ordered by timestamp

* test: sds: avoid using "as any" as it bypasses type checks

* test: filter: avoid using "as any" as it bypasses type checks

* test: fix tests without introducing proxy
2025-08-28 15:57:23 +10:00
fryorcraken
8542d04bf5
feat: query on connect (#2602)
* feat: query on connect

Perform store time-range queries upon connecting to a store node.
Some heuristics are applied to ensure the store queries are not too frequent.

* make `maybeQuery` private

* query-on-connect: use index.ts only for re-export

* query-on-connect: update doc
2025-08-28 13:58:16 +10:00
fryorcraken
826aedd558
log: logs are already prefixed with waku (#2603) 2025-08-27 23:32:41 +10:00
fryorcraken
0dfbcf6b6b
feat: StoreConnect events (#2601)
* store connect evt: use enum instead of free strings for Waku event types

* store connect evt: more accurate enum name

* store connect evt: add store connect event on peer manager

* store connect evt: simplify logic statements

* store connect evt: test store connect

* store connect evt: export event types

* test: use enum

* Shorter name for waku events
2025-08-27 12:29:22 +10:00
fryorcraken
78c856d079
fix: remove sharding circular dependency (#2590) 2025-08-25 10:49:05 +10:00
Sasha
836d6b8793
feat: expose message hash from IDecodedMessage (#2578)
* expose message hash from IDecodedMessage

* up mock

* optimize hashing and add tests
2025-08-21 11:58:43 +02:00
Sasha
c8dfdb1ace
fix: make health events emission consistent (#2570)
* fix health events emission

* fix bloom test flakiness due to bloom filter properties
2025-08-20 13:13:38 +02:00
Sasha
26de2d11c8
chore: release master (#2515) 2025-08-15 21:56:59 +02:00
Sasha
dfb2baf004
fix: prevent setting shard info from PX if it exists (#2561)
* prevent setting px shard info

* address review
2025-08-15 00:30:07 +02:00
Sasha
eab8ce81b4
feat!: local peer discovery improvements (#2557)
* update local peer discovery, make it configurable for cache

* move to separate file

* up tests, remove local storage from tests

* pass local peer cache options

* add e2e tests

* add aditional e2e tests for local cache

* rename local-peer-cache into peer-cache

* update tests, ci

* prevent filterign ws addresses
2025-08-15 00:14:32 +02:00
Sasha
95da57a870
feat: peer exchange discovery improvements (#2537)
* rename, mark stuff to remove

* rename type

* update exports from discovery package, improve PX implementation

* re-structure px discovery, comment tests

* implement UT, E2E Tests, clean code a bit, implement recurring PX discovery, move DNS E2E Tests to separate folder, remove not needed E2E tests

* fix discovery dialer e2e test

* mark as started

* fix dns tests

* attempt to fix e2e

* skip test

* update tests

* fix typo

* add catch all in stream manager, update tests

* update mock

* update test
2025-08-14 09:47:55 +02:00
fryorcraken
c161b37d08
fix!: SDS acknowledgements (#2549)
* SDS: change default causal history size to 200

# Conflicts:
#	packages/sds/src/index.ts
#	packages/sds/src/message_channel/message_channel.ts

* SDS: add some comments

* SDS: segregate messages types, introduce LocalHistory

* SDS: fix miss-acks

* SDS: logs and more explicit variable names

* SDS: shorten event name

* SDS: shorten var name

* SDS: move message classes to own file.

* SDS: use lodash instead of custom SortedArray implementation

* SDS: move Message to own file

* SDS: add comparison tests
2025-08-14 10:44:18 +10:00
fryorcraken
de972d6694
fix: reduce emission of health events (#2554) 2025-08-13 23:42:22 +10:00
Sasha
ada265731a
fix: improve error handling for stream manager (#2546)
* fix: improve error handling for stream manager

* fix browser tests

* update logs for stream manager

* fix tests

* fix playwright
2025-08-12 23:25:23 +02:00
fryorcraken
dc5155056b
feat!: SDS improvements and fixes (#2539)
* introduce `MessageId` type

# Conflicts:
#	packages/sds/src/message_channel/message_channel.ts

* fix: own messages are not used for ack

* fix: own messages are not used for ack

* doc: long term solution is SDS protocol change

* SDS: renaming to match message function

* SDS: introduce `Message` class for easier encoding/decoding

# Conflicts:
#	packages/sds/src/message_channel/events.ts
#	packages/sds/src/message_channel/message_channel.ts

* SDS Message is a class now

* SDS: it's "possibly" not "partially" acknowledged.

* SDS: TODO

* SDS: fix tests

* SDS: make logs start with `waku`

* SDS: add bloom filter test

# Conflicts:
#	packages/sds/src/message_channel/events.spec.ts

* SDS: improve naming

* SDS: improve naming

Messages are not "sent" or received, but pushed for processing in local queues.

* SDS: sync message should not be delivered

* SDS: renaming from earlier

* SDS: remove useless variable

* SDS: Fix comment

* SDS: sync messages do not get "delivered"

* SDS: acks

* SDS: simplify delivered event

* SDS: improve event naming

* SDS: fix comment

* SDS: make task error an official event

* SDS: Mark messages that are irretrievably lost

* SDS: remove default for irretrievable and simplify config

* SDS: typo on sync event

* SDS: add and user sender id

* SDS: resent message never get ack'd

* SDS: fix cylic dependencies

* SDS: helpful logs

* SDS: avoid duplicate history entries

* SDS: export options
2025-08-12 10:47:52 +10:00
fryorcraken
459fe96fe6
fix: should not self-acknowledge messages (#2528)
* introduce `MessageId` type

* fix deprecated import

* test: own messages are not used for acks

* fix: own messages are not used for ack

* fix: own messages are not used for ack

* test: do not self-possible-ack

* doc: long term solution is SDS protocol change
2025-08-08 10:18:01 +10:00
fryorcraken
52e5c34520
test: add browser tests for sds (#2527)
* tests: add browser tests for sds

* test: use FIREFOX_BIN if set
2025-08-06 10:30:00 +10:00
fryorcraken
449797d5c1
fix: bloom filter protobuf decoding (#2529)
* SDS: add bloom filter test

* fix: ensure bloom filter can be decoded from protobuf
2025-08-05 14:55:57 +10:00
fryorcraken
d7919e8c9b
improve var name 2025-07-29 19:29:21 +10:00
fryorcraken
f4a2778e02
test: return first peer without traversing branch 2025-07-29 19:29:20 +10:00
fryorcraken
25f884e05b
feat: retrieve peers from all passed enrtree URLs
Order retrieval is random, but it attemps to retrieve from all ENR trees.
2025-07-29 19:29:20 +10:00
fryorcraken
0dfe35281c
fix: do not limit DNS Peer Discovery on capability
Nodes returned by DNS Discovery are not guaranteed to be reachable.
Hence, setting an upfront limit based on sought capability does not provide
any guarantees that such capability should be reached.

The discovery's mechanism role is to find Waku 2 nodes. As many as possible.

It is then the role of the peer manager to decide:
- whether to attempt connecting to the node based on advertised capabilities
- retain connection to the node based on actual mounted protocols.

We still want to prevent infinite loops, hence the `maxGet` parameter.
Also, there was a dichotomy between code tested, and code actually used by libp2p peer discovery, now resolved.

# Conflicts:
#	packages/discovery/src/dns/constants.ts
2025-07-29 19:29:13 +10:00
fbarbu15
c1f9471cd7
chore: extra reliability tests 2 (#2450)
* chore: add throughput reliability tests

* chore: add capability to run all

* chore: add network-latency reliability tests

* chore: add network-latency reliability tests

* chore: add other network reliability tests

* chore: add other network reliability tests

* chore: fix tc cleanup

* chore: refactor common code

* chore: refactor common code

* chore: refactor common code

* chore: refactor common code

* chore: refactor common code

* chore: refactor common code

* chore: refactor common code

* chore: fix

* chore: fix tests

* chore: fix tests
2025-07-29 10:17:43 +03:00
fryorcraken
f8f20db85c
small optimization 2025-07-22 10:59:47 +10:00
fryorcraken
cd265ba1ae
fix test for nwaku master 2025-07-21 15:54:05 +10:00
fryorcraken
39c139158c
Delete dual test (makes it hard to work on nwaku master) 2025-07-21 15:44:30 +10:00
fryorcraken
77c694095d
fixing test against nwaku nightly 2025-07-21 15:40:05 +10:00
fryorcraken
158f6ecf98
fix retry_manager test 2025-07-21 15:02:26 +10:00
fryorcraken
d4429702c2
introduce interface for better mocking
As one could fall in a trap of not defining the right methods on the mock
2025-07-21 14:47:07 +10:00
fryorcraken
1e7e0291fa
separate file tests
To not have 2 `runTests` in same file.
2025-07-21 14:10:00 +10:00
fryorcraken
b51d598581
re-enable commented test 2025-07-21 13:57:06 +10:00
fryorcraken
ec8ded9c6a
revert distracting changes 2025-07-21 12:26:51 +10:00
fryorcraken
3bb5a4931f
fix peer exchange test 2025-07-21 12:26:29 +10:00
fryorcraken
0739fd1dd2
revert shard info -> relay shard renaming 2025-07-19 21:50:51 +10:00
fryorcraken
e5919a6bd9
revert most invasive changes 2025-07-19 21:25:21 +10:00
fryorcraken
97a26b2373
revert test changes 2025-07-19 19:21:39 +10:00
fryorcraken
e53717cd08
revert temporary change 2025-07-19 19:03:51 +10:00
fryorcraken
3c930dedfc
revert change on encoder 2025-07-19 18:07:38 +10:00
fryorcraken
065ef0adb0
getRoutingInfo is best as a method as it can use the context from this. 2025-07-19 17:32:58 +10:00
fryorcraken
8d278f8132
revert change that is just noise 2025-07-19 17:24:34 +10:00
fryorcraken
1bbb129d04
fix relay test 2025-07-19 17:19:58 +10:00
fryorcraken
3200b19a02
Simplify IRouting interface 2025-07-19 14:45:06 +10:00
fryorcraken
d04130feb6
use IRouting info for typing 2025-07-19 14:30:44 +10:00
fryorcraken
b8867dee38
test: re-introduce usage of ensureSubscriptions 2025-07-19 14:24:30 +10:00
fryorcraken
b4787e0e87
remove default values for cluster id and num shards in cluster
These are high level configuration values, that need to be consistent within a given execution. Setting them as default value at such a low level is a footgun risk.
2025-07-19 13:22:45 +10:00