status-im
/
consul
mirror of https://github.com/status-im/consul.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update telemetry docs (#9905) 

* Fixes #2379-Improve interval explanation in the telemetry doc

* Fixes #4734-Update consul memory metrics

* Fixes #4836-Removed node.deregistration as that isn't in state.go

* Fixes #8986 partially-Trim redundant language

* Fixes #9087-Adds helpful details to telemetry on autopilot

* Fixes #9274-Addresses NaN output in autopilot
This commit is contained in:
Jono Sosulska 2021-03-22 15:47:41 -07:00 committed by GitHub
parent eeceb85edb
commit 277de5addc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 184 additions and 169 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

353
website/content/docs/agent/telemetry.mdx
View File

@ -12,7 +12,15 @@ description: >-
The Consul agent collects various runtime metrics about the performance of
different libraries and subsystems. These metrics are aggregated on a ten
second interval and are retained for one minute.
second (10s) interval and are retained for one minute. An _interval_ is the period of time between instances of data being collected and aggregated.
When telemetry is being streamed to an external metrics store, the interval is defined to be that store's flush interval.
|External Store|Interval (seconds)|
|:--------|:--------|
|[dogstatsd](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent#how-it-works)|10s|
|[Prometheus](https://vector.dev/docs/reference/configuration/sinks/prometheus_exporter/#flush_period_secshttps://vector.dev/docs/reference/configuration/sinks/prometheus_exporter/#flush_period_secs)| 60s|
|[statsd](https://github.com/statsd/statsd/blob/master/docs/metric_types.md#timing)|10s|
To view this data, you must send a signal to the Consul process: on Unix,
this is `USR1` while on Windows it is `BREAK`. Once Consul receives the signal,
@ -60,10 +68,10 @@ These are some metrics emitted that can help you understand the health of your c
| Metric Name | Description | Unit | Type |
| :----------------------- | :----------------------------------------------------------------------------------- | :--------------------------- | :------ |
| `consul.kvs.apply` | This measures the time it takes to complete an update to the KV store. | ms | timer |
| `consul.txn.apply` | This measures the time spent applying a transaction operation. | ms | timer |
| `consul.raft.apply` | This counts the number of Raft transactions occurring over the interval. | raft transactions / interval | counter |
| `consul.raft.commitTime` | This measures the time it takes to commit a new entry to the Raft log on the leader. | ms | timer |
| `consul.kvs.apply` | Measures the time it takes to complete an update to the KV store. | ms | timer |
| `consul.txn.apply` | Measures the time spent applying a transaction operation. | ms | timer |
| `consul.raft.apply` | Counts the number of Raft transactions occurring over the interval. | raft transactions / interval | counter |
| `consul.raft.commitTime` | Measures the time it takes to commit a new entry to the Raft log on the leader. | ms | timer |
**Why they're important:** Taken together, these metrics indicate how long it takes to complete write operations in various parts of the Consul cluster. Generally these should all be fairly consistent and no more than a few milliseconds. Sudden changes in any of the timing values could be due to unexpected load on the Consul servers, or due to problems on the servers themselves.
@ -74,8 +82,8 @@ These are some metrics emitted that can help you understand the health of your c
| Metric Name | Description | Unit | Type |
| :------------------------------- | :------------------------------------------------------------------------------------------------------------- | :-------- | :------ |
| `consul.raft.leader.lastContact` | Measures the time since the leader was last able to contact the follower nodes when checking its leader lease. | ms | timer |
| `consul.raft.state.candidate` | This increments whenever a Consul server starts an election. | elections | counter |
| `consul.raft.state.leader` | This increments whenever a Consul server becomes a leader. | leaders | counter |
| `consul.raft.state.candidate` | Increments whenever a Consul server starts an election. | elections | counter |
| `consul.raft.state.leader` | Increments whenever a Consul server becomes a leader. | leaders | counter |
**Why they're important:** Normally, your Consul cluster should have a stable leader. If there are frequent elections or leadership changes, it would likely indicate network issues between the Consul servers, or that the Consul servers themselves are unable to keep up with the load.
@ -85,23 +93,34 @@ These are some metrics emitted that can help you understand the health of your c
| Metric Name | Description | Unit | Type |
| :------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------- | :---- |
| `consul.autopilot.healthy` | This tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0. | health state | gauge |
| `consul.autopilot.healthy` | Tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0. All non-leader servers will report `NaN`. | health state | gauge |
**Why it's important:** Obviously, you want your cluster to be healthy.
**Why it's important:** Autopilot can expose the overall health of your cluster with a simple boolean.
**What to look for:** Alert if `healthy` is 0.
**What to look for:** Alert if `healthy` is 0. Some other indicators of an unhealthy cluster would be:
- `consul.raft.commitTime` - This can help reflect the speed of state store
changes being performmed by the agent. If this number is rising, the server may
be experiencing an issue due to degraded resources on the host.
- [Leadership change metrics](#leadership-changes) - Check for deviation from
the recommended values. This can indicate failed leadership elections or
flapping nodes.
### Memory usage
| Metric Name | Description | Unit | Type |
| :--------------------------- | :----------------------------------------------------------------- | :---- | :---- |
| `consul.runtime.alloc_bytes` | This measures the number of bytes allocated by the Consul process. | bytes | gauge |
| `consul.runtime.sys_bytes` | This is the total number of bytes of memory obtained from the OS. | bytes | gauge |
| `consul.runtime.alloc_bytes` | Measures the number of bytes allocated by the Consul process. | bytes | gauge |
| `consul.runtime.sys_bytes` | Measures the total number of bytes of memory obtained from the OS. | bytes | gauge |
**Why they're important:** Consul keeps all of its data in memory. If Consul consumes all available memory, it will crash.
**What to look for:** If `consul.runtime.sys_bytes` exceeds 90% of total available system memory.
**NOTE:** This metric is calculated using Go's runtime package
[MemStats](https://golang.org/pkg/runtime/#MemStats). This will have a
different output than using information gathered from `top`. For more
information, see [GH-4734](https://github.com/hashicorp/consul/issues/4734).
### Garbage collection
| Metric Name | Description | Unit | Type |
@ -129,61 +148,57 @@ you will need to apply a function such as InfluxDB's [`non_negative_difference()
Sudden large changes to the `consul.client.rpc` metrics (greater than 50% deviation from baseline).
`consul.client.rpc.exceeded` or `consul.client.rpc.failed` count > 0, as it implies that an agent is being rate-limited or fails to make an RPC request to a Consul server
When telemetry is being streamed to an external metrics store, the interval is defined to
be that store's flush interval. Otherwise, the interval can be assumed to be 10 seconds
when retrieving metrics from the built-in store using the above described signals.
## Metrics Reference
This is a full list of metrics emitted by Consul.
| Metric | Description | Unit | Type |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------- |
| `consul.acl.blocked.{check,node,service}.deregistration` | This increments whenever a deregistration fails for an entity (check, node or service) is blocked by an ACL. | requests | counter |
| `consul.acl.blocked.{check,node,service}.registration` | This increments whenever a registration fails for an entity (check, node or service) is blocked by an ACL. | requests | counter |
| `consul.acl.blocked.{check,service}.deregistration` | Increments whenever a deregistration fails for an entity (check or service) is blocked by an ACL. | requests | counter |
| `consul.acl.blocked.{check,node,service}.registration` | Increments whenever a registration fails for an entity (check, node or service) is blocked by an ACL. | requests | counter |
| `consul.api.http` | Migrated from consul.http.. this samples how long it takes to service the given HTTP request for the given verb and path. Includes labels for `path` and `method`. `path` does not include details like service or key names, for these an underscore will be present as a placeholder (eg. path=`v1.kv._`) | ms | timer |
| `consul.client.rpc` | This increments whenever a Consul agent in client mode makes an RPC request to a Consul server. This gives a measure of how much a given agent is loading the Consul servers. Currently, this is only generated by agents in client mode, not Consul servers. | requests | counter |
| `consul.client.rpc.exceeded` | This increments whenever a Consul agent in client mode makes an RPC request to a Consul server gets rate limited by that agent's [`limits`](/docs/agent/options#limits) configuration. This gives an indication that there's an abusive application making too many requests on the agent, or that the rate limit needs to be increased. Currently, this only applies to agents in client mode, not Consul servers. | rejected requests | counter |
| `consul.client.rpc.failed` | This increments whenever a Consul agent in client mode makes an RPC request to a Consul server and fails. | requests | counter |
| `consul.client.api.catalog_register.` | This increments whenever a Consul agent receives a catalog register request. | requests | counter |
| `consul.client.api.success.catalog_register.` | This increments whenever a Consul agent successfully responds to a catalog register request. | requests | counter |
| `consul.client.rpc.error.catalog_register.` | This increments whenever a Consul agent receives an RPC error for a catalog register request. | errors | counter |
| `consul.client.api.catalog_deregister.` | This increments whenever a Consul agent receives a catalog deregister request. | requests | counter |
| `consul.client.api.success.catalog_deregister.` | This increments whenever a Consul agent successfully responds to a catalog deregister request. | requests | counter |
| `consul.client.rpc.error.catalog_deregister.` | This increments whenever a Consul agent receives an RPC error for a catalog deregister request. | errors | counter |
| `consul.client.api.catalog_datacenters.` | This increments whenever a Consul agent receives a request to list datacenters in the catalog. | requests | counter |
| `consul.client.api.success.catalog_datacenters.` | This increments whenever a Consul agent successfully responds to a request to list datacenters. | requests | counter |
| `consul.client.rpc.error.catalog_datacenters.` | This increments whenever a Consul agent receives an RPC error for a request to list datacenters. | errors | counter |
| `consul.client.api.catalog_nodes.` | This increments whenever a Consul agent receives a request to list nodes from the catalog. | requests | counter |
| `consul.client.api.success.catalog_nodes.` | This increments whenever a Consul agent successfully responds to a request to list nodes. | requests | counter |
| `consul.client.rpc.error.catalog_nodes.` | This increments whenever a Consul agent receives an RPC error for a request to list nodes. | errors | counter |
| `consul.client.api.catalog_services.` | This increments whenever a Consul agent receives a request to list services from the catalog. | requests | counter |
| `consul.client.api.success.catalog_services.` | This increments whenever a Consul agent successfully responds to a request to list services. | requests | counter |
| `consul.client.rpc.error.catalog_services.` | This increments whenever a Consul agent receives an RPC error for a request to list services. | errors | counter |
| `consul.client.api.catalog_service_nodes.` | This increments whenever a Consul agent receives a request to list nodes offering a service. | requests | counter |
| `consul.client.api.success.catalog_service_nodes.` | This increments whenever a Consul agent successfully responds to a request to list nodes offering a service. | requests | counter |
| `consul.client.rpc` | Increments whenever a Consul agent in client mode makes an RPC request to a Consul server. This gives a measure of how much a given agent is loading the Consul servers. Currently, this is only generated by agents in client mode, not Consul servers. | requests | counter |
| `consul.client.rpc.exceeded` | Increments whenever a Consul agent in client mode makes an RPC request to a Consul server gets rate limited by that agent's [`limits`](/docs/agent/options#limits) configuration. This gives an indication that there's an abusive application making too many requests on the agent, or that the rate limit needs to be increased. Currently, this only applies to agents in client mode, not Consul servers. | rejected requests | counter |
| `consul.client.rpc.failed` | Increments whenever a Consul agent in client mode makes an RPC request to a Consul server and fails. | requests | counter |
| `consul.client.api.catalog_register.` | Increments whenever a Consul agent receives a catalog register request. | requests | counter |
| `consul.client.api.success.catalog_register.` | Increments whenever a Consul agent successfully responds to a catalog register request. | requests | counter |
| `consul.client.rpc.error.catalog_register.` | Increments whenever a Consul agent receives an RPC error for a catalog register request. | errors | counter |
| `consul.client.api.catalog_deregister.` | Increments whenever a Consul agent receives a catalog deregister request. | requests | counter |
| `consul.client.api.success.catalog_deregister.` | Increments whenever a Consul agent successfully responds to a catalog deregister request. | requests | counter |
| `consul.client.rpc.error.catalog_deregister.` | Increments whenever a Consul agent receives an RPC error for a catalog deregister request. | errors | counter |
| `consul.client.api.catalog_datacenters.` | Increments whenever a Consul agent receives a request to list datacenters in the catalog. | requests | counter |
| `consul.client.api.success.catalog_datacenters.` | Increments whenever a Consul agent successfully responds to a request to list datacenters. | requests | counter |
| `consul.client.rpc.error.catalog_datacenters.` | Increments whenever a Consul agent receives an RPC error for a request to list datacenters. | errors | counter |
| `consul.client.api.catalog_nodes.` | Increments whenever a Consul agent receives a request to list nodes from the catalog. | requests | counter |
| `consul.client.api.success.catalog_nodes.` | Increments whenever a Consul agent successfully responds to a request to list nodes. | requests | counter |
| `consul.client.rpc.error.catalog_nodes.` | Increments whenever a Consul agent receives an RPC error for a request to list nodes. | errors | counter |
| `consul.client.api.catalog_services.` | Increments whenever a Consul agent receives a request to list services from the catalog. | requests | counter |
| `consul.client.api.success.catalog_services.` | Increments whenever a Consul agent successfully responds to a request to list services. | requests | counter |
| `consul.client.rpc.error.catalog_services.` | Increments whenever a Consul agent receives an RPC error for a request to list services. | errors | counter |
| `consul.client.api.catalog_service_nodes.` | Increments whenever a Consul agent receives a request to list nodes offering a service. | requests | counter |
| `consul.client.api.success.catalog_service_nodes.` | Increments whenever a Consul agent successfully responds to a request to list nodes offering a service. | requests | counter |
| `consul.client.api.error.catalog_service_nodes.` | Increments whenever a Consul agent receives an RPC error for request to list nodes offering a service. | requests | counter |
| `consul.client.rpc.error.catalog_service_nodes.` | This increments whenever a Consul agent receives an RPC error for a request to list nodes offering a service. | errors | counter |
| `consul.client.api.catalog_node_services.` | This increments whenever a Consul agent receives a request to list services registered in a node. | requests | counter |
| `consul.client.api.success.catalog_node_services.` | This increments whenever a Consul agent successfully responds to a request to list services in a node. | requests | counter |
| `consul.client.rpc.error.catalog_node_services.` | This increments whenever a Consul agent receives an RPC error for a request to list services in a node. | errors | counter |
| `consul.client.rpc.error.catalog_service_nodes.` | Increments whenever a Consul agent receives an RPC error for a request to list nodes offering a service. | errors | counter |
| `consul.client.api.catalog_node_services.` | Increments whenever a Consul agent receives a request to list services registered in a node. | requests | counter |
| `consul.client.api.success.catalog_node_services.` | Increments whenever a Consul agent successfully responds to a request to list services in a node. | requests | counter |
| `consul.client.rpc.error.catalog_node_services.` | Increments whenever a Consul agent receives an RPC error for a request to list services in a node. | errors | counter |
| `consul.client.api.catalog_node_service_list` | Increments whenever a Consul agent receives a request to list a node's registered services. | requests | counter |
| `consul.client.rpc.error.catalog_node_service_list` | Increments whenever a Consul agent receives an RPC error for request to list a node's registered services. | errors | counter |
| `consul.client.api.success.catalog_node_service_list` | Increments whenever a Consul agent successfully responds to a request to list a node's registered services. | requests | counter |
| `consul.client.api.catalog_gateway_services.` | This increments whenever a Consul agent receives a request to list services associated with a gateway. | requests | counter |
| `consul.client.api.success.catalog_gateway_services.` | This increments whenever a Consul agent successfully responds to a request to list services associated with a gateway. | requests | counter |
| `consul.client.rpc.error.catalog_gateway_services.` | This increments whenever a Consul agent receives an RPC error for a request to list services associated with a gateway. | errors | counter |
| `consul.runtime.num_goroutines` | This tracks the number of running goroutines and is a general load pressure indicator. This may burst from time to time but should return to a steady state value. | number of goroutines | gauge |
| `consul.runtime.alloc_bytes` | This measures the number of bytes allocated by the Consul process. This may burst from time to time but should return to a steady state value. | bytes | gauge |
| `consul.runtime.heap_objects` | This measures the number of objects allocated on the heap and is a general memory pressure indicator. This may burst from time to time but should return to a steady state value. | number of objects | gauge |
| `consul.state.nodes` | This measures the current number of nodes registered with Consul. It is only emitted by Consul servers. Added in v1.9.0. | number of objects | gauge |
| `consul.state.services` | This measures the current number of unique services registered with Consul, based on service name. It is only emitted by Consul servers. Added in v1.9.0. | number of objects | gauge |
| `consul.state.service_instances` | This measures the current number of unique service instances registered with Consul. It is only emitted by Consul servers. Added in v1.9.0. | number of objects | gauge |
| `consul.dns.stale_queries` | This increments when an agent serves a query within the allowed stale threshold. | queries | counter |
| `consul.dns.ptr_query.` | This measures the time spent handling a reverse DNS query for the given node. | ms | timer |
| `consul.dns.domain_query.` | This measures the time spent handling a domain query for the given node. | ms | timer |
| `consul.http...` | DEPRECATED IN 1.9: This tracks how long it takes to service the given HTTP request for the given verb and path. Paths do not include details like service or key names, for these an underscore will be present as a placeholder (eg. `consul.http.GET.v1.kv._`) | ms | timer |
| `consul.version | This metric measures the count of running agents. | agents | guage |
| `consul.client.api.catalog_gateway_services.` | Increments whenever a Consul agent receives a request to list services associated with a gateway. | requests | counter |
| `consul.client.api.success.catalog_gateway_services.` | Increments whenever a Consul agent successfully responds to a request to list services associated with a gateway. | requests | counter |
| `consul.client.rpc.error.catalog_gateway_services.` | Increments whenever a Consul agent receives an RPC error for a request to list services associated with a gateway. | errors | counter |
| `consul.runtime.num_goroutines` | Tracks the number of running goroutines and is a general load pressure indicator. This may burst from time to time but should return to a steady state value. | number of goroutines | gauge |
| `consul.runtime.alloc_bytes` | Measures the number of bytes allocated by the Consul process. This may burst from time to time but should return to a steady state value. | bytes | gauge |
| `consul.runtime.heap_objects` | Measures the number of objects allocated on the heap and is a general memory pressure indicator. This may burst from time to time but should return to a steady state value. | number of objects | gauge |
| `consul.state.nodes` | Measures the current number of nodes registered with Consul. It is only emitted by Consul servers. Added in v1.9.0. | number of objects | gauge |
| `consul.state.services` | Measures the current number of unique services registered with Consul, based on service name. It is only emitted by Consul servers. Added in v1.9.0. | number of objects | gauge |
| `consul.state.service_instances` | Measures the current number of unique service instances registered with Consul. It is only emitted by Consul servers. Added in v1.9.0. | number of objects | gauge |
| `consul.dns.stale_queries` | Increments when an agent serves a query within the allowed stale threshold. | queries | counter |
| `consul.dns.ptr_query.` | Measures the time spent handling a reverse DNS query for the given node. | ms | timer |
| `consul.dns.domain_query.` | Measures the time spent handling a domain query for the given node. | ms | timer |
| `consul.http...` | DEPRECATED IN 1.9: Tracks how long it takes to service the given HTTP request for the given verb and path. Paths do not include details like service or key names, for these an underscore will be present as a placeholder (eg. `consul.http.GET.v1.kv._`) | ms | timer |
| `consul.version` | Measures the count of running agents. | agents | guage |
## Server Health
@ -191,65 +206,65 @@ These metrics are used to monitor the health of the Consul servers.
| Metric | Description | Unit | Type |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ------- |
| `consul.acl.apply` | This measures the time it takes to complete an update to the ACL store. | ms | timer |
| `consul.acl.resolveTokenLegacy` | This measures the time it takes to resolve an ACL token using the legacy ACL system. | ms | timer |
| `consul.acl.ResolveToken` | This measures the time it takes to resolve an ACL token. | ms | timer |
| `consul.acl.ResolveTokenToIdentity` | This measures the time it takes to resolve an ACL token to an Identity. | ms | timer |
| `consul.acl.apply` | Measures the time it takes to complete an update to the ACL store. | ms | timer |
| `consul.acl.resolveTokenLegacy` | Measures the time it takes to resolve an ACL token using the legacy ACL system. | ms | timer |
| `consul.acl.ResolveToken` | Measures the time it takes to resolve an ACL token. | ms | timer |
| `consul.acl.ResolveTokenToIdentity` | Measures the time it takes to resolve an ACL token to an Identity. | ms | timer |
| `consul.acl.token.cache_hit` | Increments if Consul is able to resolve a token's identity, or a legacy token, from the cache. | cache read op | counter |
| `consul.acl.token.cache_miss` | Increments if Consul cannot resolve a token's identity, or a legacy token, from the cache. | cache read op | counter |
| `consul.cache.bypass` | Counts how many times a request bypassed the cache because no cache-key was provided. | counter | counter |
| `consul.cache.fetch_success` | Counts the number of successful fetches by the cache. | counter | counter |
| `consul.cache.fetch_error` | Counts the number of failed fetches by the cache. | counter | counter |
| `consul.cache.evict_expired` | Counts the number of expired entries that are evicted. | counter | counter |
| `consul.raft.fsm.snapshot` | This metric measures the time taken by the FSM to record the current state for the snapshot. | ms | timer |
| `consul.raft.fsm.apply` | This metric gives the number of logs committed since the last interval. | commit logs / interval | counter |
| `consul.raft.commitNumLogs` | This metric measures the count of logs processed for application to the FSM in a single batch. | logs | gauge |
| `consul.raft.fsm.enqueue` | This metric measures the amount of time to enqueue a batch of logs for the FSM to apply. | ms | timer |
| `consul.raft.fsm.restore` | This metric measures the time taken by the FSM to restore its state from a snapshot. | ms | timer |
| `consul.raft.snapshot.create` | This metric measures the time taken to initialize the snapshot process. | ms | timer |
| `consul.raft.snapshot.persist` | This metric measures the time taken to dump the current snapshot taken by the Consul agent to the disk. | ms | timer |
| `consul.raft.snapshot.takeSnapshot` | This metric measures the total time involved in taking the current snapshot (creating one and persisting it) by the Consul agent. | ms | timer |
| `consul.raft.replication.heartbeat` | This metric measures the time taken to invoke appendEntries on a peer, so that it doesn’t timeout on a periodic basis. | ms | timer |
| `consul.serf.snapshot.appendLine` | This metric measures the time taken by the Consul agent to append an entry into the existing log. | ms | timer |
| `consul.serf.snapshot.compact` | This metric measures the time taken by the Consul agent to compact a log. This operation occurs only when the snapshot becomes large enough to justify the compaction . | ms | timer |
| `consul.raft.fsm.snapshot` | Measures the time taken by the FSM to record the current state for the snapshot. | ms | timer |
| `consul.raft.fsm.apply` | The number of logs committed since the last interval. | commit logs / interval | counter |
| `consul.raft.commitNumLogs` | Measures the count of logs processed for application to the FSM in a single batch. | logs | gauge |
| `consul.raft.fsm.enqueue` | Measures the amount of time to enqueue a batch of logs for the FSM to apply. | ms | timer |
| `consul.raft.fsm.restore` | Measures the time taken by the FSM to restore its state from a snapshot. | ms | timer |
| `consul.raft.snapshot.create` | Measures the time taken to initialize the snapshot process. | ms | timer |
| `consul.raft.snapshot.persist` | Measures the time taken to dump the current snapshot taken by the Consul agent to the disk. | ms | timer |
| `consul.raft.snapshot.takeSnapshot` | Measures the total time involved in taking the current snapshot (creating one and persisting it) by the Consul agent. | ms | timer |
| `consul.raft.replication.heartbeat` | Measures the time taken to invoke appendEntries on a peer, so that it doesn’t timeout on a periodic basis. | ms | timer |
| `consul.serf.snapshot.appendLine` | Measures the time taken by the Consul agent to append an entry into the existing log. | ms | timer |
| `consul.serf.snapshot.compact` | Measures the time taken by the Consul agent to compact a log. This operation occurs only when the snapshot becomes large enough to justify the compaction . | ms | timer |
| `consul.raft.applied_index` | Represents the raft applied index. | index | gauge |
| `consul.raft.last_index` | Represents the raft applied index. | index | gauge |
| `consul.raft.state.leader` | This increments whenever a Consul server becomes a leader. If there are frequent leadership changes this may be indication that the servers are overloaded and aren't meeting the soft real-time requirements for Raft, or that there are networking problems between the servers. | leadership transitions / interval | counter |
| `consul.raft.state.candidate` | This increments whenever a Consul server starts an election. If this increments without a leadership change occurring it could indicate that a single server is overloaded or is experiencing network connectivity issues. | election attempts / interval | counter |
| `consul.raft.apply` | This counts the number of Raft transactions occurring over the interval, which is a general indicator of the write load on the Consul servers. | raft transactions / interval | counter |
| `consul.raft.barrier` | This metric counts the number of times the agent has started the barrier i.e the number of times it has issued a blocking call, to ensure that the agent has all the pending operations that were queued, to be applied to the agent's FSM. | blocks / interval | counter |
| `consul.raft.verify_leader` | This metric counts the number of times an agent checks whether it is still the leader or not | checks / interval | Counter |
| `consul.raft.restore` | This metric counts the number of times the restore operation has been performed by the agent. Here, restore refers to the action of raft consuming an external snapshot to restore its state. | operation invoked / interval | counter |
| `consul.raft.commitTime` | This measures the time it takes to commit a new entry to the Raft log on the leader. | ms | timer |
| `consul.raft.leader.dispatchLog` | This measures the time it takes for the leader to write log entries to disk. | ms | timer |
| `consul.raft.leader.dispatchNumLogs` | This metric measures the number of logs committed to disk in a batch. | logs | gauge |
| `consul.raft.replication.appendEntries` | This measures the time it takes to replicate log entries to followers. This is a general indicator of the load pressure on the Consul servers, as well as the performance of the communication between the servers. | ms | timer |
| `consul.raft.state.follower` | This metric counts the number of times an agent has entered the follower mode. This happens when a new agent joins the cluster or after the end of a leader election. | follower state entered / interval | counter |
| `consul.raft.transistion.heartbeat_timeout` | This metric gives the number of times an agent has transitioned to the Candidate state, after receive no heartbeat messages from the last known leader. | timeouts / interval | counter |
| `consul.raft.restoreUserSnapshot` | This metric measures the time taken by the agent to restore the FSM state from a user's snapshot | ms | timer |
| `consul.raft.rpc.processHeartBeat` | This metric measures the time taken to process a heartbeat request. | ms | timer |
| `consul.raft.rpc.appendEntries` | This metric measures the time taken to process an append entries RPC call from an agent. | ms | timer |
| `consul.raft.rpc.appendEntries.storeLogs` | This metric measures the time taken to add any outstanding logs for an agent, since the last appendEntries was invoked | ms | timer |
| `consul.raft.rpc.appendEntries.processLogs` | This metric measures the time taken to process the outstanding log entries of an agent. | ms | timer |
| `consul.raft.rpc.requestVote` | This metric measures the time taken to process the request vote RPC call. | ms | timer |
| `consul.raft.rpc.installSnapshot` | This metric measures the time taken to process the installSnapshot RPC call. This metric should only be seen on agents which are currently in the follower state. | ms | timer |
| `consul.raft.replication.appendEntries.rpc` | This metric measures the time taken by the append entries RFC, to replicate the log entries of a leader agent onto its follower agent(s) | ms | timer |
| `consul.raft.replication.appendEntries.logs` | This metric measures the number of logs replicated to an agent, to bring it up to speed with the leader's logs. | logs appended/ interval | counter |
| `consul.raft.state.leader` | Increments whenever a Consul server becomes a leader. If there are frequent leadership changes this may be indication that the servers are overloaded and aren't meeting the soft real-time requirements for Raft, or that there are networking problems between the servers. | leadership transitions / interval | counter |
| `consul.raft.state.candidate` | Increments whenever a Consul server starts an election. If this increments without a leadership change occurring it could indicate that a single server is overloaded or is experiencing network connectivity issues. | election attempts / interval | counter |
| `consul.raft.apply` | Counts the number of Raft transactions occurring over the interval, which is a general indicator of the write load on the Consul servers. | raft transactions / interval | counter |
| `consul.raft.barrier` | Counts the number of times the agent has started the barrier i.e the number of times it has issued a blocking call, to ensure that the agent has all the pending operations that were queued, to be applied to the agent's FSM. | blocks / interval | counter |
| `consul.raft.verify_leader` | Counts the number of times an agent checks whether it is still the leader or not | checks / interval | Counter |
| `consul.raft.restore` | Counts the number of times the restore operation has been performed by the agent. Here, restore refers to the action of raft consuming an external snapshot to restore its state. | operation invoked / interval | counter |
| `consul.raft.commitTime` | Measures the time it takes to commit a new entry to the Raft log on the leader. | ms | timer |
| `consul.raft.leader.dispatchLog` | Measures the time it takes for the leader to write log entries to disk. | ms | timer |
| `consul.raft.leader.dispatchNumLogs` | Measures the number of logs committed to disk in a batch. | logs | gauge |
| `consul.raft.replication.appendEntries` | Measures the time it takes to replicate log entries to followers. This is a general indicator of the load pressure on the Consul servers, as well as the performance of the communication between the servers. | ms | timer |
| `consul.raft.state.follower` | Counts the number of times an agent has entered the follower mode. This happens when a new agent joins the cluster or after the end of a leader election. | follower state entered / interval | counter |
| `consul.raft.transistion.heartbeat_timeout` | The number of times an agent has transitioned to the Candidate state, after receive no heartbeat messages from the last known leader. | timeouts / interval | counter |
| `consul.raft.restoreUserSnapshot` | Measures the time taken by the agent to restore the FSM state from a user's snapshot | ms | timer |
| `consul.raft.rpc.processHeartBeat` | Measures the time taken to process a heartbeat request. | ms | timer |
| `consul.raft.rpc.appendEntries` | Measures the time taken to process an append entries RPC call from an agent. | ms | timer |
| `consul.raft.rpc.appendEntries.storeLogs` | Measures the time taken to add any outstanding logs for an agent, since the last appendEntries was invoked | ms | timer |
| `consul.raft.rpc.appendEntries.processLogs` | Measures the time taken to process the outstanding log entries of an agent. | ms | timer |
| `consul.raft.rpc.requestVote` | Measures the time taken to process the request vote RPC call. | ms | timer |
| `consul.raft.rpc.installSnapshot` | Measures the time taken to process the installSnapshot RPC call. This metric should only be seen on agents which are currently in the follower state. | ms | timer |
| `consul.raft.replication.appendEntries.rpc` | Measures the time taken by the append entries RFC, to replicate the log entries of a leader agent onto its follower agent(s) | ms | timer |
| `consul.raft.replication.appendEntries.logs` | Measures the number of logs replicated to an agent, to bring it up to speed with the leader's logs. | logs appended/ interval | counter |
| `consul.raft.leader.lastContact` | This will only be emitted by the Raft leader and measures the time since the leader was last able to contact the follower nodes when checking its leader lease. It can be used as a measure for how stable the Raft timing is and how close the leader is to timing out its lease.The lease timeout is 500 ms times the [`raft_multiplier` configuration](/docs/agent/options#raft_multiplier), so this telemetry value should not be getting close to that configured value, otherwise the Raft timing is marginal and might need to be tuned, or more powerful servers might be needed. See the [Server Performance](/docs/install/performance) guide for more details. | ms | timer |
| `consul.rpc.accept_conn` | This increments when a server accepts an RPC connection. | connections | counter |
| `consul.catalog.register` | This measures the time it takes to complete a catalog register operation. | ms | timer |
| `consul.catalog.deregister` | This measures the time it takes to complete a catalog deregister operation. | ms | timer |
| `consul.fsm.register` | This measures the time it takes to apply a catalog register operation to the FSM. | ms | timer |
| `consul.fsm.deregister` | This measures the time it takes to apply a catalog deregister operation to the FSM. | ms | timer |
| `consul.fsm.acl.` | This measures the time it takes to apply the given ACL operation to the FSM. | ms | timer |
| `consul.fsm.session.` | This measures the time it takes to apply the given session operation to the FSM. | ms | timer |
| `consul.fsm.kvs.` | This measures the time it takes to apply the given KV operation to the FSM. | ms | timer |
| `consul.fsm.tombstone.` | This measures the time it takes to apply the given tombstone operation to the FSM. | ms | timer |
| `consul.fsm.coordinate.batch-update` | This measures the time it takes to apply the given batch coordinate update to the FSM. | ms | timer |
| `consul.fsm.prepared-query.` | This measures the time it takes to apply the given prepared query update operation to the FSM. | ms | timer |
| `consul.fsm.txn` | This measures the time it takes to apply the given transaction update to the FSM. | ms | timer |
| `consul.fsm.autopilot` | This measures the time it takes to apply the given autopilot update to the FSM. | ms | timer |
| `consul.fsm.persist` | This measures the time it takes to persist the FSM to a raft snapshot. | ms | timer |
| `consul.rpc.accept_conn` | Increments when a server accepts an RPC connection. | connections | counter |
| `consul.catalog.register` | Measures the time it takes to complete a catalog register operation. | ms | timer |
| `consul.catalog.deregister` | Measures the time it takes to complete a catalog deregister operation. | ms | timer |
| `consul.fsm.register` | Measures the time it takes to apply a catalog register operation to the FSM. | ms | timer |
| `consul.fsm.deregister` | Measures the time it takes to apply a catalog deregister operation to the FSM. | ms | timer |
| `consul.fsm.acl.` | Measures the time it takes to apply the given ACL operation to the FSM. | ms | timer |
| `consul.fsm.session.` | Measures the time it takes to apply the given session operation to the FSM. | ms | timer |
| `consul.fsm.kvs.` | Measures the time it takes to apply the given KV operation to the FSM. | ms | timer |
| `consul.fsm.tombstone.` | Measures the time it takes to apply the given tombstone operation to the FSM. | ms | timer |
| `consul.fsm.coordinate.batch-update` | Measures the time it takes to apply the given batch coordinate update to the FSM. | ms | timer |
| `consul.fsm.prepared-query.` | Measures the time it takes to apply the given prepared query update operation to the FSM. | ms | timer |
| `consul.fsm.txn` | Measures the time it takes to apply the given transaction update to the FSM. | ms | timer |
| `consul.fsm.autopilot` | Measures the time it takes to apply the given autopilot update to the FSM. | ms | timer |
| `consul.fsm.persist` | Measures the time it takes to persist the FSM to a raft snapshot. | ms | timer |
| `consul.fsm.intention` | Measures the time it takes to apply an intention operation to the state store. | ms | timer |
| `consul.fsm.ca` | Measures the time it takes to apply CA configuration operations to the FSM. | ms | timer |
| `consul.fsm.ca.leaf` | Measures the time it takes to apply an operation while signing a leaf certificate. | ms | timer |
@ -258,35 +273,35 @@ These metrics are used to monitor the health of the Consul servers.
| `consul.fsm.acl.bindingrule` | Measures the time it takes to apply an ACL binding rule operation to the FSM. | ms | timer |
| `consul.fsm.acl.authmethod` | Measures the time it takes to apply an ACL authmethod operation to the FSM. | ms | timer |
| `consul.fsm.system_metadata` | Measures the time it takes to apply a system metadata operation to the FSM. | ms | timer |
| `consul.kvs.apply` | This measures the time it takes to complete an update to the KV store. | ms | timer |
| `consul.leader.barrier` | This measures the time spent waiting for the raft barrier upon gaining leadership. | ms | timer |
| `consul.leader.reconcile` | This measures the time spent updating the raft store from the serf member information. | ms | timer |
| `consul.leader.reconcileMember` | This measures the time spent updating the raft store for a single serf member's information. | ms | timer |
| `consul.leader.reapTombstones` | This measures the time spent clearing tombstones. | ms | timer |
| `consul.prepared-query.apply` | This measures the time it takes to apply a prepared query update. | ms | timer |
| `consul.prepared-query.explain` | This measures the time it takes to process a prepared query explain request. | ms | timer |
| `consul.prepared-query.execute` | This measures the time it takes to process a prepared query execute request. | ms | timer |
| `consul.prepared-query.execute_remote` | This measures the time it takes to process a prepared query execute request that was forwarded to another datacenter. | ms | timer |
| `consul.rpc.raft_handoff` | This increments when a server accepts a Raft-related RPC connection. | connections | counter |
| `consul.rpc.request_error` | This increments when a server returns an error from an RPC request. | errors | counter |
| `consul.rpc.request` | This increments when a server receives a Consul-related RPC request. | requests | counter |
| `consul.rpc.query` | This increments when a server receives a new blocking RPC request, indicating the rate of new blocking query calls. See consul.rpc.queries_blocking for the current number of in-flight blocking RPC calls. This metric changed in 1.7.0 to only increment on the the start of a query. The rate of queries will appear lower, but is more accurate. | queries | counter |
| `consul.rpc.queries_blocking` | This shows the current number of in-flight blocking queries the server is handling. | queries | gauge |
| `consul.rpc.cross-dc` | This increments when a server sends a (potentially blocking) cross datacenter RPC query. | queries | counter |
| `consul.rpc.consistentRead` | This measures the time spent confirming that a consistent read can be performed. | ms | timer |
| `consul.session.apply` | This measures the time spent applying a session update. | ms | timer |
| `consul.session.renew` | This measures the time spent renewing a session. | ms | timer |
| `consul.session_ttl.invalidate` | This measures the time spent invalidating an expired session. | ms | timer |
| `consul.txn.apply` | This measures the time spent applying a transaction operation. | ms | timer |
| `consul.txn.read` | This measures the time spent returning a read transaction. | ms | timer |
| `consul.grpc.client.request.count` | This metric counts the number of gRPC requests made by the client agent to a Consul server. | requests | counter |
| `consul.grpc.client.connection.count` | This metric counts the number of new gRPC connections opened by the client agent to a Consul server. | connections | counter |
| `consul.grpc.client.connections` | This metric measures the number of active gRPC connections open from the client agent to any Consul servers. | connections | gauge |
| `consul.grpc.server.request.count` | This metric counts the number of gRPC requests received by the server. | requests | counter |
| `consul.grpc.server.connection.count` | This metric counts the number of new gRPC connections received by the server. | connections | counter |
| `consul.grpc.server.connections` | This metric measures the number of active gRPC connections open on the server. | connections | gauge |
| `consul.grpc.server.stream.count` | This metric counts the number of new gRPC streams received by the server. | streams | counter |
| `consul.grpc.server.streams` | This metric measures the number of active gRPC streams handled by the server. | streams | guage |
| `consul.kvs.apply` | Measures the time it takes to complete an update to the KV store. | ms | timer |
| `consul.leader.barrier` | Measures the time spent waiting for the raft barrier upon gaining leadership. | ms | timer |
| `consul.leader.reconcile` | Measures the time spent updating the raft store from the serf member information. | ms | timer |
| `consul.leader.reconcileMember` | Measures the time spent updating the raft store for a single serf member's information. | ms | timer |
| `consul.leader.reapTombstones` | Measures the time spent clearing tombstones. | ms | timer |
| `consul.prepared-query.apply` | Measures the time it takes to apply a prepared query update. | ms | timer |
| `consul.prepared-query.explain` | Measures the time it takes to process a prepared query explain request. | ms | timer |
| `consul.prepared-query.execute` | Measures the time it takes to process a prepared query execute request. | ms | timer |
| `consul.prepared-query.execute_remote` | Measures the time it takes to process a prepared query execute request that was forwarded to another datacenter. | ms | timer |
| `consul.rpc.raft_handoff` | Increments when a server accepts a Raft-related RPC connection. | connections | counter |
| `consul.rpc.request_error` | Increments when a server returns an error from an RPC request. | errors | counter |
| `consul.rpc.request` | Increments when a server receives a Consul-related RPC request. | requests | counter |
| `consul.rpc.query` | Increments when a server receives a new blocking RPC request, indicating the rate of new blocking query calls. See consul.rpc.queries_blocking for the current number of in-flight blocking RPC calls. This metric changed in 1.7.0 to only increment on the the start of a query. The rate of queries will appear lower, but is more accurate. | queries | counter |
| `consul.rpc.queries_blocking` | The current number of in-flight blocking queries the server is handling. | queries | gauge |
| `consul.rpc.cross-dc` | Increments when a server sends a (potentially blocking) cross datacenter RPC query. | queries | counter |
| `consul.rpc.consistentRead` | Measures the time spent confirming that a consistent read can be performed. | ms | timer |
| `consul.session.apply` | Measures the time spent applying a session update. | ms | timer |
| `consul.session.renew` | Measures the time spent renewing a session. | ms | timer |
| `consul.session_ttl.invalidate` | Measures the time spent invalidating an expired session. | ms | timer |
| `consul.txn.apply` | Measures the time spent applying a transaction operation. | ms | timer |
| `consul.txn.read` | Measures the time spent returning a read transaction. | ms | timer |
| `consul.grpc.client.request.count` | Counts the number of gRPC requests made by the client agent to a Consul server. | requests | counter |
| `consul.grpc.client.connection.count` | Counts the number of new gRPC connections opened by the client agent to a Consul server. | connections | counter |
| `consul.grpc.client.connections` | Measures the number of active gRPC connections open from the client agent to any Consul servers. | connections | gauge |
| `consul.grpc.server.request.count` | Counts the number of gRPC requests received by the server. | requests | counter |
| `consul.grpc.server.connection.count` | Counts the number of new gRPC connections received by the server. | connections | counter |
| `consul.grpc.server.connections` | Measures the number of active gRPC connections open on the server. | connections | gauge |
| `consul.grpc.server.stream.count` | Counts the number of new gRPC streams received by the server. | streams | counter |
| `consul.grpc.server.streams` | Measures the number of active gRPC streams handled by the server. | streams | guage |
## Cluster Health
@ -294,38 +309,38 @@ These metrics give insight into the health of the cluster as a whole.
| Metric | Description | Unit | Type |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ------- |
| `consul.memberlist.degraded.probe` | This metric counts the number of times the agent has performed failure detection on another agent at a slower probe rate. The agent uses its own health metric as an indicator to perform this action. (If its health score is low, means that the node is healthy, and vice versa.) | probes / interval | counter |
| `consul.memberlist.degraded.timeout` | This metric counts the number of times an agent was marked as a dead node, whilst not getting enough confirmations from a randomly selected list of agent nodes in an agent's membership. | occurrence / interval | counter |
| `consul.memberlist.msg.dead` | This metric counts the number of times an agent has marked another agent to be a dead node. | messages / interval | counter |
| `consul.memberlist.health.score` | This metric describes a node's perception of its own health based on how well it is meeting the soft real-time requirements of the protocol. This metric ranges from 0 to 8, where 0 indicates "totally healthy". This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals. For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf | score | gauge |
| `consul.memberlist.msg.suspect` | This increments when an agent suspects another as failed when executing random probes as part of the gossip protocol. These can be an indicator of overloaded agents, network problems, or configuration errors where agents can not connect to each other on the [required ports](/docs/agent/options#ports). | suspect messages received / interval | counter |
| `consul.memberlist.tcp.accept` | This metric counts the number of times an agent has accepted an incoming TCP stream connection. | connections accepted / interval | counter |
| `consul.memberlist.udp.sent/received` | This metric measures the total number of bytes sent/received by an agent through the UDP protocol. | bytes sent or bytes received / interval | counter |
| `consul.memberlist.tcp.connect` | This metric counts the number of times an agent has initiated a push/pull sync with an other agent. | push/pull initiated / interval | counter |
| `consul.memberlist.tcp.sent` | This metric measures the total number of bytes sent by an agent through the TCP protocol | bytes sent / interval | counter |
| `consul.memberlist.gossip` | This metric measures the time taken for gossip messages to be broadcasted to a set of randomly selected nodes. | ms | timer |
| `consul.memberlist.msg_alive` | This metric counts the number of alive messages, that the agent has processed so far, based on the message information given by the network layer. | messages / Interval | counter |
| `consul.memberlist.msg_dead` | This metric gives the number of dead messages, that the agent has processed so far, based on the message information given by the network layer. | messages / Interval | counter |
| `consul.memberlist.msg_suspect` | This metric gives the number of suspect messages, that the agent has processed so far, based on the message information given by the network layer. | messages / Interval | counter |
| `consul.memberlist.probeNode` | This metric measures the time taken to perform a single round of failure detection on a select agent. | nodes / Interval | counter |
| `consul.memberlist.pushPullNode` | This metric measures the number of agents that have exchanged state with this agent. | nodes / Interval | counter |
| `consul.serf.member.failed` | This increments when an agent is marked dead. This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the [required ports](/docs/agent/options#ports). | failures / interval | counter |
| `consul.memberlist.degraded.probe` | Counts the number of times the agent has performed failure detection on another agent at a slower probe rate. The agent uses its own health metric as an indicator to perform this action. (If its health score is low, means that the node is healthy, and vice versa.) | probes / interval | counter |
| `consul.memberlist.degraded.timeout` | Counts the number of times an agent was marked as a dead node, whilst not getting enough confirmations from a randomly selected list of agent nodes in an agent's membership. | occurrence / interval | counter |
| `consul.memberlist.msg.dead` | Counts the number of times an agent has marked another agent to be a dead node. | messages / interval | counter |
| `consul.memberlist.health.score` | Describes a node's perception of its own health based on how well it is meeting the soft real-time requirements of the protocol. This metric ranges from 0 to 8, where 0 indicates "totally healthy". This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals. For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf | score | gauge |
| `consul.memberlist.msg.suspect` | Increments when an agent suspects another as failed when executing random probes as part of the gossip protocol. These can be an indicator of overloaded agents, network problems, or configuration errors where agents can not connect to each other on the [required ports](/docs/agent/options#ports). | suspect messages received / interval | counter |
| `consul.memberlist.tcp.accept` | Counts the number of times an agent has accepted an incoming TCP stream connection. | connections accepted / interval | counter |
| `consul.memberlist.udp.sent/received` | Measures the total number of bytes sent/received by an agent through the UDP protocol. | bytes sent or bytes received / interval | counter |
| `consul.memberlist.tcp.connect` | Counts the number of times an agent has initiated a push/pull sync with an other agent. | push/pull initiated / interval | counter |
| `consul.memberlist.tcp.sent` | Measures the total number of bytes sent by an agent through the TCP protocol | bytes sent / interval | counter |
| `consul.memberlist.gossip` | Measures the time taken for gossip messages to be broadcasted to a set of randomly selected nodes. | ms | timer |
| `consul.memberlist.msg_alive` | Counts the number of alive messages, that the agent has processed so far, based on the message information given by the network layer. | messages / Interval | counter |
| `consul.memberlist.msg_dead` | The number of dead messages that the agent has processed so far, based on the message information given by the network layer. | messages / Interval | counter |
| `consul.memberlist.msg_suspect` | The number of suspect messages that the agent has processed so far, based on the message information given by the network layer. | messages / Interval | counter |
| `consul.memberlist.probeNode` | Measures the time taken to perform a single round of failure detection on a select agent. | nodes / Interval | counter |
| `consul.memberlist.pushPullNode` | Measures the number of agents that have exchanged state with this agent. | nodes / Interval | counter |
| `consul.serf.member.failed` | Increments when an agent is marked dead. This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the [required ports](/docs/agent/options#ports). | failures / interval | counter |
| `consul.serf.member.flap` | Available in Consul 0.7 and later, this increments when an agent is marked dead and then recovers within a short time period. This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the [required ports](/docs/agent/options#ports). | flaps / interval | counter |
| `consul.serf.member.join` | This increments when an agent joins the cluster. If an agent flapped or failed this counter also increments when it re-joins. | joins / interval | counter |
| `consul.serf.member.left` | This increments when an agent leaves the cluster. | leaves / interval | counter |
| `consul.serf.events` | This increments when an agent processes an [event](/commands/event). Consul uses events internally so there may be additional events showing in telemetry. There are also a per-event counters emitted as `consul.serf.events.`. | events / interval | counter |
| `consul.serf.member.join` | Increments when an agent joins the cluster. If an agent flapped or failed this counter also increments when it re-joins. | joins / interval | counter |
| `consul.serf.member.left` | Increments when an agent leaves the cluster. | leaves / interval | counter |
| `consul.serf.events` | Increments when an agent processes an [event](/commands/event). Consul uses events internally so there may be additional events showing in telemetry. There are also a per-event counters emitted as `consul.serf.events.`. | events / interval | counter |
| `consul.serf.msgs.sent` | This metric is sample of the number of bytes of messages broadcast to the cluster. In a given time interval, the sum of this metric is the total number of bytes sent and the count is the number of messages sent. | message bytes / interval | counter |
| `consul.autopilot.failure_tolerance` | This tracks the number of voting servers that the cluster can lose while continuing to function. | servers | gauge |
| `consul.autopilot.healthy` | This tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0. | boolean | gauge |
| `consul.session_ttl.active` | This tracks the active number of sessions being tracked. | sessions | gauge |
| `consul.catalog.service.query.` | This increments for each catalog query for the given service. | queries | counter |
| `consul.catalog.service.query-tag..` | This increments for each catalog query for the given service with the given tag. | queries | counter |
| `consul.catalog.service.query-tags..` | This increments for each catalog query for the given service with the given tags. | queries | counter |
| `consul.catalog.service.not-found.` | This increments for each catalog query where the given service could not be found. | queries | counter |
| `consul.catalog.connect.query.` | This increments for each connect-based catalog query for the given service. | queries | counter |
| `consul.catalog.connect.query-tag..` | This increments for each connect-based catalog query for the given service with the given tag. | queries | counter |
| `consul.catalog.connect.query-tags..` | This increments for each connect-based catalog query for the given service with the given tags. | queries | counter |
| `consul.catalog.connect.not-found.` | This increments for each connect-based catalog query where the given service could not be found. | queries | counter |
| `consul.autopilot.failure_tolerance` | Tracks the number of voting servers that the cluster can lose while continuing to function. | servers | gauge |
| `consul.autopilot.healthy` | Tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0. All non-leader servers will report `NaN`. | boolean | gauge |
| `consul.session_ttl.active` | Tracks the active number of sessions being tracked. | sessions | gauge |
| `consul.catalog.service.query.` | Increments for each catalog query for the given service. | queries | counter |
| `consul.catalog.service.query-tag..` | Increments for each catalog query for the given service with the given tag. | queries | counter |
| `consul.catalog.service.query-tags..` | Increments for each catalog query for the given service with the given tags. | queries | counter |
| `consul.catalog.service.not-found.` | Increments for each catalog query where the given service could not be found. | queries | counter |
| `consul.catalog.connect.query.` | Increments for each connect-based catalog query for the given service. | queries | counter |
| `consul.catalog.connect.query-tag..` | Increments for each connect-based catalog query for the given service with the given tag. | queries | counter |
| `consul.catalog.connect.query-tags..` | Increments for each connect-based catalog query for the given service with the given tags. | queries | counter |
| `consul.catalog.connect.not-found.` | Increments for each connect-based catalog query where the given service could not be found. | queries | counter |
## Connect Built-in Proxy Metrics
@ -362,8 +377,8 @@ agent. The table below describes the additional metrics exported by the proxy.
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------- |
| `consul.proxy.web.runtime.*` | The same go runtime metrics as documented for the agent above. | mixed | mixed |
| `consul.proxy.web.inbound.conns` | Shows the current number of connections open from inbound requests to the proxy. Where supported a `dst` label is added indicating the service name the proxy represents. | connections | gauge |
| `consul.proxy.web.inbound.rx_bytes` | This increments by the number of bytes received from an inbound client connection. Where supported a `dst` label is added indicating the service name the proxy represents. | bytes | counter |
| `consul.proxy.web.inbound.tx_bytes` | This increments by the number of bytes transferred to an inbound client connection. Where supported a `dst` label is added indicating the service name the proxy represents. | bytes | counter |
| `consul.proxy.web.inbound.rx_bytes` | Increments by the number of bytes received from an inbound client connection. Where supported a `dst` label is added indicating the service name the proxy represents. | bytes | counter |
| `consul.proxy.web.inbound.tx_bytes` | Increments by the number of bytes transferred to an inbound client connection. Where supported a `dst` label is added indicating the service name the proxy represents. | bytes | counter |
| `consul.proxy.web.upstream.conns` | Shows the current number of connections open from a proxy instance to an upstream. Where supported a `src` label is added indicating the service name the proxy represents, and a `dst` label is added indicating the service name the upstream is connecting to. | connections | gauge |
| `consul.proxy.web.inbound.rx_bytes` | This increments by the number of bytes received from an upstream connection. Where supported a `src` label is added indicating the service name the proxy represents, and a `dst` label is added indicating the service name the upstream is connecting to. | bytes | counter |
| `consul.proxy.web.inbound.tx_bytes` | This increments by the number of bytes transferred to an upstream connection. Where supported a `src` label is added indicating the service name the proxy represents, and a `dst` label is added indicating the service name the upstream is connecting to. | bytes | counter |
| `consul.proxy.web.inbound.rx_bytes` | Increments by the number of bytes received from an upstream connection. Where supported a `src` label is added indicating the service name the proxy represents, and a `dst` label is added indicating the service name the upstream is connecting to. | bytes | counter |
| `consul.proxy.web.inbound.tx_bytes` | Increments by the number of bytes transferred to an upstream connection. Where supported a `src` label is added indicating the service name the proxy represents, and a `dst` label is added indicating the service name the upstream is connecting to. | bytes | counter |