Pop Chunhapanya b421b3ab05
GossipSub v1.2: IDONTWANT control message and priority queue. (#553)
## GossipSub v1.2 implementation

Specification: libp2p/specs#548

### Work Summary
Sending IDONTWANT

Implement a smart queue
Add priorities to the smart queue

    Put IDONTWANT packets into the smart priority queue as soon as the node gets the packets

Handling IDONTWANT

Use a map to remember the message ids whose IDONTWANT packets have been received
Implement max_idontwant_messages (ignore the IDONWANT packets if the max is reached)
Clear the message IDs from the cache after 3 heartbeats

    Hash the message IDs before putting them into the cache.

More requested features

    Add a feature test to not send IDONTWANT if the other side doesnt support it

### Commit Summary

* Replace sending channel with the smart rpcQueue

Since we want to implement a priority queue later, we need to replace
the normal sending channels with the new smart structures first.

* Implement UrgentPush in the smart rpcQueue

UrgentPush allows you to push an rpc packet to the front of the queue so
that it will be popped out fast.

* Add IDONTWANT to rpc.proto and trace.proto

* Send IDONTWANT right before validation step

Most importantly, this commit adds a new method called PreValidation to
the interface PubSubRouter, which will be called right before validating
the gossipsub message.

In GossipSubRouter, PreValidation will send the IDONTWANT controll
messages to all the mesh peers of the topics of the received messages.

* Test GossipSub IDONWANT sending

* Send IDONWANT only for large messages

* Handle IDONTWANT control messages

When receiving IDONTWANTs, the host should remember the message ids
contained in IDONTWANTs using a hash map.

When receiving messages with those ids, it shouldn't forward them to the
peers who already sent the IDONTWANTs.

When the maximum number of IDONTWANTs is reached for any particular
peer, the host should ignore any excessive IDONTWANTs from that peer.

* Clear expired message IDs from the IDONTWANT cache

If the messages IDs received from IDONTWANTs are older than 3
heartbeats, they should be removed from the IDONTWANT cache.

* Keep the hashes of IDONTWANT message ids instead

Rather than keeping the raw message ids, keep their hashes instead to
save memory and protect again memory DoS attacks.

* Increase GossipSubMaxIHaveMessages to 1000

* fixup! Clear expired message IDs from the IDONTWANT cache

* Not send IDONTWANT if the receiver doesn't support

* fixup! Replace sending channel with the smart rpcQueue

* Not use pointers in rpcQueue

* Simply rcpQueue by using only one mutex

* Check ctx error in rpc sending worker

Co-authored-by: Steven Allen <steven@stebalien.com>

* fixup! Simply rcpQueue by using only one mutex

* fixup! Keep the hashes of IDONTWANT message ids instead

* Use AfterFunc instead implementing our own

* Fix misc lint errors

* fixup! Fix misc lint errors

* Revert "Increase GossipSubMaxIHaveMessages to 1000"

This reverts commit 6fabcdd068a5f5238c5280a3460af9c3998418ec.

* Increase GossipSubMaxIDontWantMessages to 1000

* fixup! Handle IDONTWANT control messages

* Skip TestGossipsubConnTagMessageDeliveries

* Skip FuzzAppendOrMergeRPC

* Revert "Skip FuzzAppendOrMergeRPC"

This reverts commit f141e13234de0960d139339acb636a1afea9e219.

* fixup! Send IDONWANT only for large messages

* fixup! fixup! Keep the hashes of IDONTWANT message ids instead

* fixup! Implement UrgentPush in the smart rpcQueue

* fixup! Use AfterFunc instead implementing our own

---------

Co-authored-by: Steven Allen <steven@stebalien.com>
2024-08-16 18:16:35 +03:00
2020-10-08 20:18:21 +03:00
2024-08-06 19:52:49 +00:00
2024-08-06 19:52:49 +00:00
2019-01-11 12:17:48 +02:00
2024-08-06 19:52:49 +00:00
2024-05-20 17:13:30 -07:00
2024-05-20 17:13:30 -07:00
2020-03-25 08:42:05 +00:00
2020-03-25 08:42:05 +00:00
2020-03-25 08:42:05 +00:00
2020-03-25 08:42:05 +00:00
2022-01-23 09:29:32 +02:00
2022-08-26 02:45:41 -07:00
2022-01-23 09:29:32 +02:00
2024-05-20 16:20:16 -07:00
2022-03-03 21:18:37 +02:00
2024-05-20 16:20:16 -07:00
2022-08-26 02:45:41 -07:00
2024-08-06 19:52:49 +00:00
2021-06-15 13:00:02 +03:00
2024-08-06 19:52:49 +00:00
2024-08-06 19:52:49 +00:00
2024-08-06 19:52:49 +00:00
2024-08-06 19:52:49 +00:00
2023-03-01 16:15:54 +02:00

go-libp2p-pubsub


This repo contains the canonical pubsub implementation for libp2p. We currently provide three message router options:

  • Floodsub, which is the baseline flooding protocol.
  • Randomsub, which is a simple probabilistic router that propagates to random subsets of peers.
  • Gossipsub, which is a more advanced router with mesh formation and gossip propagation. See spec and implementation for more details.

Repo Lead Maintainer

@vyzo

This repo follows the Repo Lead Maintainer Protocol

Table of Contents

Install

go get github.com/libp2p/go-libp2p-pubsub

Usage

To be used for messaging in p2p instrastructure (as part of libp2p) such as IPFS, Ethereum, other blockchains, etc.

Example

https://github.com/libp2p/go-libp2p/tree/master/examples/pubsub

Documentation

See the libp2p specs for high level documentation and godoc for API documentation.

In this repo, you will find

.
├── LICENSE
├── README.md
# Regular Golang repo set up
├── codecov.yml
├── pb
├── go.mod
├── go.sum
├── doc.go
# PubSub base
├── pubsub.go
├── blacklist.go
├── notify.go
├── comm.go
├── discovery.go
├── sign.go
├── subscription.go
├── topic.go
├── trace.go
├── tracer.go
├── validation.go
# Floodsub router
├── floodsub.go
# Randomsub router
├── randomsub.go
# Gossipsub router
├── gossipsub.go
├── score.go
├── score_params.go
└── mcache.go

Tracing

The pubsub system supports tracing, which collects all events pertaining to the internals of the system. This allows you to recreate the complete message flow and state of the system for analysis purposes.

To enable tracing, instantiate the pubsub system using the WithEventTracer option; the option accepts a tracer with three available implementations in-package (trace to json, pb, or a remote peer). If you want to trace using a remote peer, you can do so using the traced daemon from go-libp2p-pubsub-tracer. The package also includes a utility program, tracestat, for analyzing the traces collected by the daemon.

For instance, to capture the trace as a json file, you can use the following option:

tracer, err := pubsub.NewJSONTracer("/path/to/trace.json")
if err != nil {
  panic(err)
}

pubsub.NewGossipSub(..., pubsub.WithEventTracer(tracer))

To capture the trace as a protobuf, you can use the following option:

tracer, err := pubsub.NewPBTracer("/path/to/trace.pb")
if err != nil {
  panic(err)
}

pubsub.NewGossipSub(..., pubsub.WithEventTracer(tracer))

Finally, to use the remote tracer, you can use the following incantations:

// assuming that your tracer runs in x.x.x.x and has a peer ID of QmTracer
pi, err := peer.AddrInfoFromP2pAddr(ma.StringCast("/ip4/x.x.x.x/tcp/4001/p2p/QmTracer"))
if err != nil {
  panic(err)
}

tracer, err := pubsub.NewRemoteTracer(ctx, host, pi)
if err != nil {
  panic(err)
}

ps, err := pubsub.NewGossipSub(..., pubsub.WithEventTracer(tracer))

Contribute

Contributions welcome. Please check out the issues.

Check out our contributing document for more information on how we work, and about contributing in general. Please be aware that all interactions related to multiformats are subject to the IPFS Code of Conduct.

Small note: If editing the README, please conform to the standard-readme specification.

License

The go-libp2p-pubsub project is dual-licensed under Apache 2.0 and MIT terms:

Description
Languages
Go 100%