mirror of https://github.com/status-im/consul.git
new docs for consul and consul-k8s troubleshoot command (#16284)
* new docs for consul and consul-k8s troubleshoot command * add changelog * add troubleshoot command * address comments, and update cli output to match * revert changes to troubleshoot upstreams, changes will happen in separate pr * Update .changelog/16284.txt Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com> * address comments * update trouble proxy output * add missing s, add required fields in usage --------- Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>
This commit is contained in:
malizz
GitHub
parent
085c0addc0
commit
c9c49ea3a2
|
|
@ -0,0 +1,3 @@
|
||||||
|
```release-note:feature
|
||||||
|
cli: adds new CLI commands `consul troubleshoot upstreams` and `consul troubleshoot proxy` to troubleshoot Consul's service mesh configuration and network issues.
|
||||||
|
```
|
||||||
|
|
@ -56,6 +56,7 @@ Available commands are:
|
||||||
services Interact with services
|
services Interact with services
|
||||||
snapshot Saves, restores and inspects snapshots of Consul server state
|
snapshot Saves, restores and inspects snapshots of Consul server state
|
||||||
tls Builtin helpers for creating CAs and certificates
|
tls Builtin helpers for creating CAs and certificates
|
||||||
|
troubleshoot Provides tools to troubleshoot Consul's service mesh configuration
|
||||||
validate Validate config files/directories
|
validate Validate config files/directories
|
||||||
version Prints the Consul version
|
version Prints the Consul version
|
||||||
watch Watch for changes in Consul
|
watch Watch for changes in Consul
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
---
|
||||||
|
layout: commands
|
||||||
|
page_title: 'Commands: Troubleshoot'
|
||||||
|
description: >-
|
||||||
|
The `consul troubleshoot` command provides tools to troubleshoot Consul's service mesh configuration.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Consul Troubleshooting
|
||||||
|
|
||||||
|
Command: `consul troubleshoot`
|
||||||
|
|
||||||
|
Use the `troubleshoot` command to diagnose Consul service mesh configuration or network issues.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```text
|
||||||
|
Usage: consul troubleshoot <subcommand> [options]
|
||||||
|
|
||||||
|
# ...
|
||||||
|
|
||||||
|
Subcommands:
|
||||||
|
|
||||||
|
proxy Troubleshoots service mesh issues from the current Envoy instance
|
||||||
|
upstreams Gets upstream Envoy identifiers and IPs configured for the proxy
|
||||||
|
```
|
||||||
|
|
||||||
|
For more information, examples, and usage about a subcommand, click on the name
|
||||||
|
of the subcommand in the sidebar or one of the links below:
|
||||||
|
|
||||||
|
- [proxy](/consul/commands/troubleshoot/proxy)
|
||||||
|
- [upstreams](/consul/commands/troubleshoot/upstreams)
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
---
|
||||||
|
layout: commands
|
||||||
|
page_title: 'Commands: Troubleshoot Proxy'
|
||||||
|
description: >-
|
||||||
|
The `consul troubleshoot proxy` diagnoses Consul service mesh configuration and network issues to an upstream.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Consul Troubleshoot Proxy
|
||||||
|
|
||||||
|
Command: `consul troubleshoot proxy`
|
||||||
|
|
||||||
|
The `troubleshoot proxy` command diagnoses Consul service mesh configuration and network issues to an upstream.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Usage: `consul troubleshoot proxy (-upstream-ip <ip>|-upstream-envoy-id <envoy-id>) [options]`
|
||||||
|
This command requires `-envoy-admin-endpoint` or `-upstream-ip` to specify the upstream.
|
||||||
|
|
||||||
|
#### Command Options
|
||||||
|
|
||||||
|
- `-envoy-admin-endpoint=<value>` - Envoy admin endpoint address for the local Envoy instance.
|
||||||
|
Defaults to `127.0.0.1:19000`.
|
||||||
|
|
||||||
|
- `-upstream-ip=<value>` - The IP address of the upstream service; Use when the upstream is reached via a transparent proxy DNS address (KubeDNS or Consul DNS)
|
||||||
|
|
||||||
|
- `-upstream-envoy-id=<value>` - The Envoy identifier of the upstream service; Use when the upstream is explicitly configured on the downstream service.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
The following example illustrates how to troubleshoot Consul service mesh configuration and network issues to an upstream from a source proxy.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul troubleshoot proxy -upstream-ip 10.4.6.160
|
||||||
|
|
||||||
|
==> Validation
|
||||||
|
✓ Certificates are valid
|
||||||
|
✓ Envoy has 0 rejected configurations
|
||||||
|
✓ Envoy has detected 0 connection failure(s)
|
||||||
|
✓ Listener for upstream "backend" found
|
||||||
|
✓ Route for upstream "backend" found
|
||||||
|
✓ Cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
|
||||||
|
✓ Healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
|
||||||
|
✓ Cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
|
||||||
|
! No healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
|
||||||
|
-> Check that your upstream service is healthy and running
|
||||||
|
-> Check that your upstream service is registered with Consul
|
||||||
|
-> Check that the upstream proxy is healthy and running
|
||||||
|
-> If you are explicitly configuring upstreams, ensure the name of the upstream is correct
|
||||||
|
```
|
||||||
|
|
||||||
|
The following example troubleshoots Consul service mesh configuration and network issues from the local Envoy instance to the given upstream.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot proxy -upstream-envoy-id db
|
||||||
|
|
||||||
|
==> Validation
|
||||||
|
✓ Certificates are valid
|
||||||
|
✓ Envoy has 0 rejected configurations
|
||||||
|
✓ Envoy has detected 0 connection failure(s)
|
||||||
|
✓ Listener for upstream "db" found
|
||||||
|
✓ Route for upstream "db" found
|
||||||
|
✓ Cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0.consul" for upstream "db" found
|
||||||
|
✓ Healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0.consul" for upstream "db" found
|
||||||
|
|
||||||
|
If you are still experiencing issues, you can:
|
||||||
|
-> Check intentions to ensure the upstream allows traffic from this source
|
||||||
|
-> If using transparent proxy, ensure DNS resolution is to the same IP you have verified here
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
---
|
||||||
|
layout: commands
|
||||||
|
page_title: 'Commands: Troubleshoot Upstreams'
|
||||||
|
description: >-
|
||||||
|
The `consul troubleshoot upstreams` lists the available upstreams in the Consul service mesh from the current service.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Consul Troubleshoot Upstreams
|
||||||
|
|
||||||
|
Command: `consul troubleshoot upstreams`
|
||||||
|
|
||||||
|
The `troubleshoot upstreams` lists the available upstreams in the Consul service mesh from the current service.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Usage: `consul troubleshoot upstreams [options]`
|
||||||
|
|
||||||
|
#### Command Options
|
||||||
|
|
||||||
|
- `-envoy-admin-endpoint=<value>` - Envoy admin endpoint address for the local Envoy instance.
|
||||||
|
Defaults to `127.0.0.1:19000`.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
Display all transparent proxy upstreams in Consul service mesh from the current Envoy instance.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul troubleshoot upstreams
|
||||||
|
==> Upstreams (explicit upstreams only) (0)
|
||||||
|
|
||||||
|
==> Upstreams IPs (transparent proxy only) (1)
|
||||||
|
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
|
||||||
|
|
||||||
|
If you cannot find the upstream address or cluster for a transparent proxy upstream:
|
||||||
|
- Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
|
||||||
|
- To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.
|
||||||
|
```
|
||||||
|
|
@ -33,6 +33,7 @@ You can use the following commands with `consul-k8s`.
|
||||||
- [`proxy list`](#proxy-list): List all Pods running proxies managed by Consul.
|
- [`proxy list`](#proxy-list): List all Pods running proxies managed by Consul.
|
||||||
- [`proxy read`](#proxy-read): Inspect the Envoy configuration for a given Pod.
|
- [`proxy read`](#proxy-read): Inspect the Envoy configuration for a given Pod.
|
||||||
- [`status`](#status): Check the status of a Consul installation on Kubernetes.
|
- [`status`](#status): Check the status of a Consul installation on Kubernetes.
|
||||||
|
- [`troubleshoot`](#troubleshoot): Troubleshoot Consul service mesh and networking issues from a given pod.
|
||||||
- [`uninstall`](#uninstall): Uninstall Consul deployment.
|
- [`uninstall`](#uninstall): Uninstall Consul deployment.
|
||||||
- [`upgrade`](#upgrade): Upgrade Consul on Kubernetes from an existing installation.
|
- [`upgrade`](#upgrade): Upgrade Consul on Kubernetes from an existing installation.
|
||||||
- [`version`](#version): Print the version of the Consul on Kubernetes CLI.
|
- [`version`](#version): Print the version of the Consul on Kubernetes CLI.
|
||||||
|
|
@ -490,6 +491,106 @@ $ consul-k8s status
|
||||||
✓ Consul clients healthy (3/3)
|
✓ Consul clients healthy (3/3)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### `troubleshoot`
|
||||||
|
|
||||||
|
The `troubleshoot` command exposes two subcommands for troubleshooting Consul
|
||||||
|
service mesh and network issues from a given pod.
|
||||||
|
|
||||||
|
- [`troubleshoot upstreams`](#troubleshoot-upstreams): List all Envoy upstreams in Consul service mesh from the given pod.
|
||||||
|
- [`troubleshoot proxy`](#troubleshoot-proxy): Troubleshoot Consul service mesh configuration and network issues between the given pod and the given upstream.
|
||||||
|
|
||||||
|
### `troubleshoot upstreams`
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot upstreams -pod <pod> <OPTIONS>
|
||||||
|
```
|
||||||
|
|
||||||
|
| Flag | Description | Default |
|
||||||
|
| ------------------------------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| <nobr>`-namespace`, `-n`</nobr> | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. |
|
||||||
|
| <nobr>`-envoy-admin-endpoint`</nobr> | `String` Envoy sidecar address and port | `127.0.0.1:19000` |
|
||||||
|
|
||||||
|
#### Example Commands
|
||||||
|
|
||||||
|
The following example displays all transparent proxy upstreams in Consul service mesh from the given pod.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot upstreams -pod frontend-767ccfc8f9-6f6gx
|
||||||
|
|
||||||
|
==> Upstreams (explicit upstreams only) (0)
|
||||||
|
|
||||||
|
==> Upstreams IPs (transparent proxy only) (1)
|
||||||
|
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
|
||||||
|
|
||||||
|
If you cannot find the upstream address or cluster for a transparent proxy upstream:
|
||||||
|
- Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
|
||||||
|
- To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.
|
||||||
|
```
|
||||||
|
|
||||||
|
The following example displays all explicit upstreams from the given pod in the Consul service mesh.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot upstreams -pod client-767ccfc8f9-6f6gx
|
||||||
|
|
||||||
|
==> Upstreams (explicit upstreams only) (1)
|
||||||
|
server
|
||||||
|
counting
|
||||||
|
|
||||||
|
==> Upstreams IPs (transparent proxy only) (0)
|
||||||
|
|
||||||
|
If you cannot find the upstream address or cluster for a transparent proxy upstream:
|
||||||
|
- Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
|
||||||
|
- To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.
|
||||||
|
```
|
||||||
|
|
||||||
|
### `troubleshoot proxy`
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot proxy -pod <pod> -upstream-ip <ip> <OPTIONS>
|
||||||
|
$ consul-k8s troubleshoot proxy -pod <pod> -upstream-envoy-id <envoy-id> <OPTIONS>
|
||||||
|
```
|
||||||
|
|
||||||
|
| Flag | Description | Default |
|
||||||
|
| ------------------------------------ | ----------------------------------------------------------| ---------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| <nobr>`-namespace`, `-n`</nobr> | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. |
|
||||||
|
| <nobr>`-envoy-admin-endpoint`</nobr> | `String` Envoy sidecar address and port | `127.0.0.1:19000` |
|
||||||
|
| <nobr>`-upstream-ip`</nobr> | `String` The IP address of the upstream transparent proxy | |
|
||||||
|
| <nobr>`-upstream-envoy-id`</nobr> | `String` The Envoy identifier of the upstream | |
|
||||||
|
|
||||||
|
#### Example Commands
|
||||||
|
|
||||||
|
The following example troubleshoots the Consul service mesh configuration and network issues between the given pod and the given upstream IP.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-ip 10.4.6.160
|
||||||
|
|
||||||
|
==> Validation
|
||||||
|
✓ certificates are valid
|
||||||
|
✓ Envoy has 0 rejected configurations
|
||||||
|
✓ Envoy has detected 0 connection failure(s)
|
||||||
|
✓ listener for upstream "backend" found
|
||||||
|
✓ route for upstream "backend" found
|
||||||
|
✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found
|
||||||
|
✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
|
||||||
|
✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found
|
||||||
|
! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
|
||||||
|
```
|
||||||
|
|
||||||
|
The following example troubleshoots the Consul service mesh configuration and network issues between the given pod and the given upstream.
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-envoy-id db
|
||||||
|
|
||||||
|
==> Validation
|
||||||
|
✓ certificates are valid
|
||||||
|
✓ Envoy has 0 rejected configurations
|
||||||
|
✓ Envoy has detected 0 connection failure(s)
|
||||||
|
! no listener for upstream "db" found
|
||||||
|
! no route for upstream "backend" found
|
||||||
|
! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
|
||||||
|
! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
|
||||||
|
```
|
||||||
|
|
||||||
### `uninstall`
|
### `uninstall`
|
||||||
|
|
||||||
The `uninstall` command removes Consul from Kubernetes.
|
The `uninstall` command removes Consul from Kubernetes.
|
||||||
|
|
|
||||||
|
|
@ -528,6 +528,23 @@
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"title": "troubleshoot",
|
||||||
|
"routes": [
|
||||||
|
{
|
||||||
|
"title": "Overview",
|
||||||
|
"path": "troubleshoot"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "upstreams",
|
||||||
|
"path": "troubleshoot/upstreams"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "proxy",
|
||||||
|
"path": "troubleshoot/proxy"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"title": "validate",
|
"title": "validate",
|
||||||
"path": "validate"
|
"path": "validate"
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue