status-im
/
status-go
mirror of https://github.com/status-im/status-go.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Handle peer event async in metrics 

There might be an issue on how we handle metrics, which causes the p2p
server to hang.

updateNodeMetrics calls  a method on the p2p server, which
blocks until the server is available:

e60f425b45/vendor/github.com/ethereum/go-ethereum/p2p/server.go (L301)
e60f425b45/vendor/github.com/ethereum/go-ethereum/p2p/server.go (L746)
If there's back-pressure on the peer event feed
e60f425b45/vendor/github.com/ethereum/go-ethereum/p2p/server.go (L783)

The event channel above might become while updateNodeMetrics
is called, which means is never consumed, the server blocks on publishing on
it, and the two will deadlock (server waits for the channel above to be consumed,
this code waits for the server to respond to peerCount, which is in the same
event loop).

Calling it in a different go-routine will allow this code to keep
processing peer added events, therefore the server will not lock and keep processing requests.
This commit is contained in:
Andrea Maria Piana 2021-02-01 10:44:51 +01:00
parent e60f425b45
commit b4e5bf417b
1 changed files with 20 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
metrics/node/subscribe.go
View File

@ -31,9 +31,26 @@ func SubscribeServerEvents(ctx context.Context, node *node.Node) error {
select {
case event := <-ch:
if isAddDropPeerEvent(event.Type) {
if err := updateNodeMetrics(node, event.Type); err != nil {
logger.Error("failed to update node metrics", "err", err)
}
// We start a goroutine here because updateNodeMetrics
// is calling a method on the p2p server, which
// blocks until the server is available:
// https://github.com/status-im/status-go/blob/e60f425b45d00d3880b42fdd77b460ec465a9f55/vendor/github.com/ethereum/go-ethereum/p2p/server.go#L301
// https://github.com/status-im/status-go/blob/e60f425b45d00d3880b42fdd77b460ec465a9f55/vendor/github.com/ethereum/go-ethereum/p2p/server.go#L746
// If there's back-pressure on the peer event feed
// https://github.com/status-im/status-go/blob/e60f425b45d00d3880b42fdd77b460ec465a9f55/vendor/github.com/ethereum/go-ethereum/p2p/server.go#L783
// The event channel above might become while updateNodeMetrics
// is called, which means is never consumed, the server blocks on publishing on
// it, and the two will deadlock (server waits for the channel above to be consumed,
// this code waits for the server to respond to peerCount, which is in the same
// event loop).
// Calling it in a different go-routine will allow this code to keep
// processing peer added events, therefore the server will not lock and
// keep processing requests.
go func() {
if err := updateNodeMetrics(node, event.Type); err != nil {
logger.Error("failed to update node metrics", "err", err)
}
}()
}
case err := <-subscription.Err():
if err != nil {