consul/website/content/docs/k8s/connect/observability/metrics.mdx

---
layout: docs
page_title: Metrics
description: Metrics for Consul on Kubernetes
---

# Metrics

Consul on Kubernetes integrates with Prometheus and Grafana to provide metrics for Consul Service Mesh. The metrics
available are:

* Connect Service metrics
* Sidecar proxy metrics
* Consul agent metrics
* Ingress, Terminating and Mesh Gateway metrics

Specific sidecar proxy metrics can also be seen in the Consul UI Topology Visualization view. This section documents how to enable each of these.

-> **Note:** Metrics will be supported in Consul-helm >= `0.31.0` and consul-k8s >= `0.25.0`. However, enabling the [metrics merging feature](#connect-service-and-sidecar-metrics-with-metrics-merging) with Helm value (`defaultEnableMerging`) or
annotation (`consul.hashicorp.com/enable-metrics-merging`) can only be used with Consul `1.10.0-alpha1` and above. The
other metrics configuration can still be used before Consul `1.10.0-alpha1`.

## Connect Service and Sidecar Metrics with Metrics Merging

Prometheus annotations are used to instruct Prometheus to scrape metrics from Pods. Prometheus annotations only support
scraping from one endpoint on a Pod, so Consul on Kubernetes supports metrics merging whereby service metrics and
sidecar proxy metrics are merged into one endpoint. If there are no service metrics, it also supports just scraping the
sidecar proxy metrics.

The diagram below illustrates how the metrics integration works when merging is enabled:

[![Metrics Merging Architecture](/img/metrics-arch.png)](/img/metrics-arch.png)

Connect service metrics can be configured with the Helm values nested under [`connectInject.metrics`](/docs/k8s/helm#v-connectinject-metrics).

Metrics and metrics merging can be enabled by default for all connect-injected Pods with the following Helm values:
```yaml
connectInject:
  metrics:
    defaultEnabled: true # by default, this inherits from the value global.metrics.enabled
    defaultEnableMerging: true
```
They can also be overridden on a per-Pod basis using the annotations `consul.hashicorp.com/enable-metrics` and
`consul.hashicorp.com/enable-metrics-merging`.

~> In most cases, the default settings will be sufficient. If you are encountering issues with colliding ports or service
metrics not being merged, you may need to change the defaults.

The Prometheus annotations configure the endpoint to scrape the metrics from. As shown in the diagram, the annotations point to a listener on `0.0.0.0:20200` on the Envoy sidecar. This listener and the corresponding Prometheus annotations can be configured with the following Helm values (or overridden on a per-Pod basis with Consul annotations `consul.hashicorp.com/prometheus-scrape-port` and `consul.hashicorp.com/prometheus-scrape-path`):
```yaml
connectInject:
  metrics:
    defaultPrometheusScrapePort: 20200
    defaultPrometheusScrapePath: "/metrics"
```
Those Helm values will result in the following Prometheus annotations being automatically added to the Pod for scraping:
```yaml
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "20200"
```

When metrics alone are enabled, the listener in the diagram on `0.0.0.0:20200` would point directly at the sidecar
metrics endpoint, rather than the merged metrics endpoint. The Prometheus scraping annotations would stay the same.

When metrics and metrics merging are *both* enabled, metrics are combined from the service and the sidecar proxy, and
exposed via a local server on the Consul sidecar for scraping. This endpoint is called the merged metrics endpoint and
defaults to `127.0.0.1:20100/stats/prometheus`. The listener will target the merged metrics endpoint in the above case.
It can be configured with the following Helm values (or overridden on a per-Pod basis with
`consul.hashicorp.com/merged-metrics-port`):
```yaml
connectInject:
  metrics:
    defaultMergedMetricsPort: 20100
```

The endpoint to scrape service metrics from can be configured only on a per-Pod basis via the Pod annotations `consul.hashicorp.com/service-metrics-port` and `consul.hashicorp.com/service-metrics-path`. If these are not configured, the service metrics port will default to the port used to register the service with Consul (`consul.hashicorp.com/connect-service-port`), which in turn defaults to the first port on the first container of the Pod. The service metrics path will default to `/metrics`.

## Consul Agent Metrics

Metrics from the Consul server and client Pods can be scraped via Prometheus by setting the field `global.metrics.enableAgentMetrics` to `true`. Additionally, one can configure the metrics retention time on the agents by configuring
the field `global.metrics.agentMetricsRetentionTime` which expects a duration and defaults to `"1m"`. This value must be greater than `"0m"` for the Consul servers and clients to emit metrics at all. As the Prometheus deployment currently does not support scraping TLS endpoints, agent metrics are currently *unsupported* when TLS is enabled.

```yaml
global:
  metrics:
    enabled: true
    enableAgentMetrics: true
    agentMetricsRetentionTime: "1m"
```

## Gateway Metrics

Metrics from the Consul gateways, namely the Ingress Gateways, Terminating Gateways and the Mesh Gateways can be scraped
via Prometheus by setting the field `global.metrics.enableGatewayMetrics` to `true`. The gateways emit standard Envoy proxy
metrics. To ensure that the metrics are not exposed to the public internet (as Mesh and Ingress gateways can have public
IPs), their metrics endpoints are exposed on the Pod IP of the respective gateway instance, rather than on all
interfaces on `0.0.0.0`.

```yaml
global:
  metrics:
    enabled: true
    enableGatewayMetrics: true
```

## Metrics in the UI Topology Visualization

Consul’s built-in UI has a topology visualization for services part of the Consul Service Mesh. The topology visualization has the ability to fetch basic metrics from a metrics provider for each service and display those metrics as part of the [topology visualization](/docs/connect/observability/ui-visualization).

The diagram below illustrates how the UI displays service metrics for a sample application:

[![UI Topology View](/img/ui-topology.png)](/img/ui-topology.png)

The topology view is configured under `ui.metrics`. This will enable the Consul UI to query the provider specified by
`ui.metrics.provider` at the URL of the Prometheus server `ui.metrics.baseURL` to display sidecar proxy metrics for the
service. The UI will display some specific sidecar proxy Prometheus metrics when `ui.metrics.enabled` is `true` and
`ui.enabled` is true. The value of `ui.metrics.enabled` defaults to `"-"` which means it will inherit from the value of
`global.metrics.enabled.`

```yaml
ui:
  enabled: true
  metrics:
    enabled: true # by default, this inherits from the value global.metrics.enabled
    provider: "prometheus"
    baseURL: http://prometheus-server
```

## Deploying Prometheus (_for demo and non-production use-cases only_)

The Helm chart contains demo manifests for deploying Prometheus. It can be installed with Helm via `prometheus.enabled`. This manifest is based on the community manifest for Prometheus.
The Prometheus deployment is designed to allow quick bootstrapping for trial and demo use cases, and is not recommended for production use-cases.

Prometheus will be installed in the same namespace as Consul, and will be installed
and uninstalled along with the Consul installation.

Grafana can optionally be utilized with Prometheus to display metrics. The installation and configuration of Grafana must be managed separately from the Consul Helm chart. The [Layer 7 Observability with Prometheus, Grafana, and Kubernetes](https://learn.hashicorp.com/tutorials/consul/kubernetes-layer7-observability?in=consul/kubernetes)) tutorial provides an installation walkthrough using Helm.

```yaml
prometheus:
  enabled: true
```