The PubSub implementation for go-libp2p https://github.com/libp2p/specs/tree/master/pubsub
Go to file
Piotr Dyraga 499109b165
Configurable size of validate queue
validateWorker() reads from validateQ and invokes validate function
that performs validation of the message. Signature validation is performed
synchronously. The number of validate workers defaults to the number of CPUs
and can be updated with WithValidateWorkers function. With no additional user
validators, signature validation is the bottleneck when receiving new messages.

Increasing the number of validating workers does not help given the context
switching and bottleneck nature of this spot. As stated in WithValidateWorkers
documentation, this function should be used rather to limit the number of workers
to devote less CPU time for synchronous validation. On the other hand, with the
default size of `validateQ`, some applications built on a top of libp2p may
experience throttled validation and lost messages.

This problem is addressed by WithValidateQueueSize allowing to configure the buffer
size for synchronous validation. Application developers knowing the nature of their
protocols can set this value to minimise the possibility of throttled synchronous
validation and dropped messages. Configurable buffer size allows to gracefully
handle peaks of messages and, from the other side, the number of concurrent
synchronous workers is still limited by validateWorkers property so the receiver
should not get congested.
2020-01-21 10:19:42 +01:00
pb chore(dep): update 2019-12-02 13:07:46 -05:00
.gitignore gx publish v0.5.0 2016-09-14 15:27:34 -07:00
.travis.yml ci: bump to go 1.13 2019-12-06 16:33:11 -05:00
LICENSE use timecache 2016-09-10 20:47:12 -07:00
README.md README: document tracing (#239) 2019-12-02 20:03:54 +00:00
blacklist.go migrate to consolidated types. (#185) 2019-05-26 17:19:03 +01:00
blacklist_test.go migrate to consolidated types. (#185) 2019-05-26 17:19:03 +01:00
codecov.yml Update Travis CI to use standard test script and go 1.9.x. 2017-11-17 22:36:56 -05:00
comm.go interface: New Topic + TopicEventHandlers objects. 2019-10-31 15:37:44 -04:00
discovery.go Added libp2p discovery capabilities to PubSub. 2019-10-31 16:39:18 -04:00
discovery_test.go fixed closed Topic handles still being able to perform some actions on the topic 2019-11-02 19:28:44 -04:00
doc.go godoc is not html either 2019-01-11 12:17:48 +02:00
floodsub.go remove useless nil check when initializing subsystem tracers 2019-11-15 20:42:04 +02:00
floodsub_test.go interface: New Topic + TopicEventHandlers objects. 2019-10-31 15:37:44 -04:00
go.mod Bump github.com/libp2p/go-libp2p-core from 0.2.5 to 0.3.0 (#245) 2019-12-16 23:56:26 +01:00
go.sum Bump github.com/libp2p/go-libp2p-core from 0.2.5 to 0.3.0 (#245) 2019-12-16 23:56:26 +01:00
gossipsub.go PR feedback: name change and account for different options order 2019-12-16 12:46:30 +01:00
gossipsub_test.go Added libp2p discovery capabilities to PubSub. 2019-10-31 16:39:18 -04:00
mcache.go PR feedback: name change and account for different options order 2019-12-16 12:46:30 +01:00
mcache_test.go fixes #247: implement msg id function as pubsub option 2019-12-16 03:54:40 +01:00
notify.go migrate to consolidated types. (#185) 2019-05-26 17:19:03 +01:00
pubsub.go rename VaidatorData to ValidatorData 2019-12-19 15:15:45 -08:00
randomsub.go remove useless nil check when initializing subsystem tracers 2019-11-15 20:42:04 +02:00
sign.go migrate to consolidated types. (#185) 2019-05-26 17:19:03 +01:00
sign_test.go migrate to consolidated types. (#185) 2019-05-26 17:19:03 +01:00
subscription.go interface: New Topic + TopicEventHandlers objects. 2019-10-31 15:37:44 -04:00
topic.go add ValidatorData field to Message 2019-11-15 20:16:58 +02:00
topic_test.go fixed closed Topic handles still being able to perform some actions on the topic 2019-11-02 19:28:44 -04:00
trace.go fixes #247: implement msg id function as pubsub option 2019-12-16 03:54:40 +01:00
tracer.go nits and beauty 2019-11-19 02:28:23 +02:00
validation.go Configurable size of validate queue 2020-01-21 10:19:42 +01:00

README.md

go-libp2p-pubsub

Discourse posts

A pubsub system with flooding and gossiping variants.

This is the canonical pubsub implementation for libp2p.

We currently provide three implementations:

  • floodsub, which is the baseline flooding protocol.
  • gossipsub, which is a more advanced router with mesh formation and gossip propagation. See spec and implementation for more details.
  • randomsub, which is a simple probabilistic router that propagates to random subsets of peers.

Table of Contents

Install

go get github.com/libp2p/go-libp2p-pubsub

Usage

To be used for messaging in p2p instrastructure (as part of libp2p) such as IPFS, Ethereum, other blockchains, etc.

Implementations

See libp2p/specs/pubsub#Implementations.

Documentation

See the libp2p specs for high level documentation and godoc for API documentation.

Tracing

The pubsub system supports tracing, which collects all events pertaining to the internals of the system. This allows you to recreate the complete message flow and state of the system for analysis purposes.

To enable tracing, instantiate the pubsub system using the WithEventTracer option; the option accepts a tracer with three available implementations in-package (trace to json, pb, or a remote peer). If you want to trace using a remote peer, you can do so using the traced daemon from go-libp2p-pubsub-tracer. The package also includes a utility program, tracestat, for analyzing the traces collected by the daemon.

For instance, to capture the trace as a json file, you can use the following option:

pubsub.NewGossipSub(..., pubsub.NewEventTracer(pubsub.NewJSONTracer("/path/to/trace.json")))

To capture the trace as a protobuf, you can use the following option:

pubsub.NewGossipSub(..., pubsub.NewEventTracer(pubsub.NewPBTracer("/path/to/trace.pb")))

Finally, to use the remote tracer, you can use the following incantations:

// assuming that your tracer runs in x.x.x.x and has a peer ID of QmTracer
pi, err := peer.AddrInfoFromP2pAddr(ma.StringCast("/ip4/x.x.x.x/tcp/4001/p2p/QmTracer"))
if err != nil {
  panic(err)
}

tracer, err := pubsub.NewRemoteTracer(ctx, host, pi)
if err != nil {
  panic(err)
}

ps, err := pubsub.NewGossipSub(..., pubsub.WithEventTracer(tracer))

Contribute

Contributions welcome. Please check out the issues.

Check out our contributing document for more information on how we work, and about contributing in general. Please be aware that all interactions related to multiformats are subject to the IPFS Code of Conduct.

Small note: If editing the README, please conform to the standard-readme specification.

License

MIT © Jeromy Johnson


The last gx published version of this module was: 0.11.16: QmfB4oDUTiaGEqT13P1JqCEhqW7cB1wpKtq3PP4BN8PhQd