mirror of https://github.com/status-im/consul.git
278 lines
13 KiB
Plaintext
278 lines
13 KiB
Plaintext
---
|
|
layout: api
|
|
page_title: Consistency Modes
|
|
description: >-
|
|
Every Consul HTTP API read request uses one of several "consistency modes" that offer
|
|
different levels of response consistency and performance. By default, Consul uses a
|
|
consistency mode that is suitable for most use cases. Users can override consistency modes
|
|
to fine-tune these trade-offs for their own use case.
|
|
---
|
|
|
|
# Consistency Modes
|
|
|
|
Every Consul HTTP API read request uses one of several _consistency modes_ that offer
|
|
different levels of response consistency and performance. By default, Consul uses a
|
|
consistency mode that is suitable for most use cases. Users can override consistency modes
|
|
to fine-tune these trade-offs for their own use case at two levels:
|
|
- Per interface (HTTP API, DNS)
|
|
- Per request (HTTP API requests only)
|
|
|
|
## What is Consistency?
|
|
|
|
Consul servers are responsible for maintaining state information like the registration and
|
|
health status of services and nodes. To protect this state against the potential failure of
|
|
Consul servers, this state is replicated across three or more Consul servers in production using a
|
|
[consensus protocol](/docs/architecture/consensus).
|
|
|
|
One Consul server is elected leader and acts as the ultimate authority on Consul's state.
|
|
If a majority of Consul servers agree to a state change, the leader is responsible for recording
|
|
the change and then distributing it to the non-leader Consul servers—known as followers.
|
|
Because it takes time for a state change on the leader to propagate to the followers,
|
|
followers may have a slightly outdated, or "stale", view of Consul's state.
|
|
|
|
If a read request is handled by the current leader, the response is guaranteed to be
|
|
fully _consistent_ (as up-to-date as possible).
|
|
If the same request were handled by a follower, the response may be less consistent:
|
|
based on a _stale_ (outdated) copy of the leader's state.
|
|
Consistency is highest if the response comes from the leader.
|
|
But ensuring only the leader can respond to the request prevents spreading
|
|
read request load across all Consul servers.
|
|
|
|
The consistency mode controls which Consul servers can repond to read requests,
|
|
enabling control over this inherent trade-off between consistency and performance.
|
|
|
|
## Available Consistency Modes
|
|
|
|
Each HTTP API endpoint documents its support for the three read consistency modes:
|
|
|
|
- `stale` -
|
|
[Consul DNS queries use `stale` mode by default](#consul-dns-queries).
|
|
This mode allows any server to handle the read regardless of whether it is the leader.
|
|
The trade-off is very fast and scalable reads with a higher likelihood of stale values.
|
|
Results are generally consistent to within 50 milliseconds of the leader,
|
|
though there is no upper limit on this staleness.
|
|
Since this mode allows reads without a leader,
|
|
a cluster that is unavailable (no quorum) can still respond to queries.
|
|
|
|
- `default` -
|
|
[Consul HTTP API queries use `default` mode by default](#consul-http-api-queries).
|
|
It is strongly consistent in almost all cases.
|
|
However, there is a small window in which a new leader may be elected
|
|
during which the old leader may respond with stale values.
|
|
The trade-off is fast reads but potentially stale values.
|
|
The condition resulting in stale reads is hard to trigger,
|
|
and most clients should not need to worry about this case.
|
|
Also, note that this race condition only applies to reads, not writes.
|
|
|
|
- `consistent` -
|
|
This mode is strongly consistent without caveats.
|
|
It requires that a leader verify with a quorum of peers that it is still leader.
|
|
This introduces an additional round-trip to all server nodes.
|
|
The trade-off is increased latency due to an extra round trip.
|
|
Most clients should not use this unless they cannot tolerate a stale read.
|
|
|
|
~> **Scaling read requests**: The most effective way to increase read scalability
|
|
is to convert non-`stale` reads to `stale` reads. If most requests are already
|
|
`stale` reads and additional load reduction is desired, use Consul Enterprise
|
|
[redundancy zones](/docs/enterprise/redundancy) or
|
|
[read replicas](/docs/enterprise/read-scale)
|
|
to spread `stale` reads across additional, _non-voting_ Consul servers.
|
|
Non-voting servers enhance read scalability without increasing the number
|
|
of voting servers; adding more then 5 voting servers is not recommended because
|
|
it decreases Consul's performance.
|
|
|
|
### Intra-Datacenter Request Behavior
|
|
|
|
The following diagrams show how the intra-datacenter request path varies
|
|
per consistency mode and the relative trade-offs between level of consistency and performance.
|
|
|
|
<Tabs>
|
|
<Tab heading="Stale reads">
|
|
|
|
[](/img/consistency-modes/consistency-mode-stale.png)
|
|
|
|
</Tab>
|
|
|
|
<Tab heading="Default reads">
|
|
|
|
[](/img/consistency-modes/consistency-mode-default.png)
|
|
|
|
</Tab>
|
|
<Tab heading="Consistent reads, all writes">
|
|
|
|
[](/img/consistency-modes/consistency-mode-consistent.png)
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
### Cross-Datacenter Request Behavior
|
|
|
|
When making a request across federated Consul datacenters, requests are forwarded from
|
|
a local server to any remote server. Once in the remote datecenter, the request path
|
|
is the same as a [local request with the same consistency mode](#intra-datacenter-request-behavior).
|
|
The following diagrams show the cross-datacenter request paths when Consul servers in datacenters are
|
|
[federated either directly or via mesh gateways](/docs/connect/gateways/mesh-gateway/wan-federation-via-mesh-gateways).
|
|
|
|
<Tabs>
|
|
<Tab heading="Traditional WAN federation">
|
|
|
|
[](/img/consistency-modes/consistency-mode-cross-datacenter-federation-traditional.png)
|
|
|
|
</Tab>
|
|
|
|
<Tab heading="WAN federation via mesh gateways">
|
|
|
|
[](/img/consistency-modes/consistency-mode-cross-datacenter-federation-mesh-gateways.png)
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
## Usage
|
|
|
|
### Consul DNS Queries
|
|
|
|
When DNS queries are issued to [Consul's DNS interface](/docs/discovery/dns),
|
|
Consul uses the `stale` consistency mode by default when interfacing with its
|
|
underlying Consul service discovery HTTP APIs
|
|
([Catalog](/api-docs/catalog), [Health](/api-docs/health), and [Prepared Query](/api-docs/query)).
|
|
|
|
The consistency mode underlying Consul DNS queries cannot be overridden on a
|
|
per-request basis.
|
|
|
|
#### Changing the Default Consistency Mode
|
|
|
|
If stronger consistency is required at the expense of increased latency,
|
|
set [`dns_config.allow_stale`] to `false` to make Consul use the
|
|
`default` consistency mode instead of `stale`
|
|
when interfacing with the underlying Consul service discovery HTTP APIs.
|
|
|
|
#### Limiting Staleness (Advanced Usage)
|
|
|
|
~> **Caution**:
|
|
Configuring [`dns_config.max_stale`] may result in a prolonged outage
|
|
if your Consul servers become overloaded.
|
|
We do not recommend using this setting to limit staleness
|
|
unless it would be preferable for your services to be unavailable
|
|
than to receive results that are too stale.
|
|
If bounded result consistency is required by a service, consider
|
|
modifying the service to use `consistent` service discovery HTTP API queries
|
|
instead of DNS lookups.
|
|
|
|
If Consul is configured to respond to DNS queries in `stale` consistency mode
|
|
([`dns_config.allow_stale`] is `true`),
|
|
you can set an upper limit on how stale a response is allowed to be by
|
|
configuring [`dns_config.max_stale`].
|
|
When the Consul agent handling the response is behind the leader by more than [`dns_config.max_stale`] time,
|
|
the request will be forwarded to the leader for handling as in `default` consistency mode.
|
|
|
|
A common cause for response staleness exceeding [`dns_config.max_stale`] is that
|
|
Consul servers are overloaded by requests. Under these circumstances, [`dns_config.max_stale`]
|
|
will make the overloading worse by converting cheap `stale` reads to more expensive `default` reads.
|
|
|
|
### Consul HTTP API Queries
|
|
|
|
All HTTP API read requests use the `default` consistency mode by default
|
|
unless [overridden on a per-request basis](#overriding-a-request-s-consistency-mode).
|
|
We do [not recommend changing the consistency mode used by default](#changing-the-default-consistency-mode-advanced-usage).
|
|
|
|
Write requests do not support defining a consistency mode.
|
|
All writes follow the behavior of the `consistent` consistency mode.
|
|
|
|
#### Overriding a Request's Consistency Mode
|
|
|
|
The HTTP API documentation for each endpoint lists which consistency modes can be used
|
|
to override the default on a per-request basis. To use a supported consistency mode
|
|
other than the default, include the desired consistency mode as a URL query parameter
|
|
when calling the endpoint:
|
|
- `stale`: Use the `stale` query parameter
|
|
- `consistent`: Use the `consistent` query parameter
|
|
- `default`: Use the `leader` query parameter;
|
|
only relevant [if the default consistency mode is changed to `stale`](#changing-the-default-consistency-mode-advanced-usage)
|
|
|
|
#### Changing the Default Consistency Mode (Advanced Usage)
|
|
|
|
~> **Caution**:
|
|
Configuring [`discovery_max_stale`] may result in a prolonged outage
|
|
if your Consul servers become overloaded.
|
|
We do not recommend using this setting to limit staleness
|
|
unless it would be preferable for your services to be unavailable
|
|
than to receive results that are too stale.
|
|
If bounded result consistency is required by a service, consider
|
|
modifying the service to use `consistent` mode
|
|
[on a per-request basis](#overriding-a-request-s-consistency-mode).
|
|
|
|
The default consistency mode can be changed to `stale` for service discovery HTTP API endpoints,
|
|
including:
|
|
- [Catalog API](/api-docs/catalog)
|
|
- [Health API](/api-docs/health)
|
|
- [Prepared Query API](/api-docs/query)
|
|
|
|
This allows Consul operators to spread service discovery read load across Consul servers
|
|
with a centralized configuration change, avoiding the need to modify every service to
|
|
explicitly set `stale` consistency mode in their requests.
|
|
Services can still override this default on a per-request basis by
|
|
[specifying a supported consistency mode as a query parameter in the request](#overriding-a-request-s-consistency-mode).
|
|
|
|
To configure Consul servers that receive service discovery requests to use `stale`
|
|
consistency mode unless overriden,
|
|
set [`discovery_max_stale`] to a value greater than zero in their agent configuration.
|
|
The `stale` consistency mode will be used by default unless the data is sufficiently stale:
|
|
its Raft log's index is more than [`discovery_max_stale`] indices behind the leader's.
|
|
Setting a positive value for [`discovery_max_stale`]
|
|
is analogous to [limiting the staleness of Consul DNS queries](#limiting-staleness-advanced-usage)
|
|
by setting [`dns_config.max_stale`].
|
|
|
|
A common cause for response staleness exceeding [`discovery_max_stale`] is that
|
|
Consul servers are overloaded by requests. Under these circumstances, [`discovery_max_stale`]
|
|
will make the overloading worse by converting cheap `stale` reads to more expensive `default` reads.
|
|
|
|
### Visibility into Response Staleness
|
|
|
|
To support bounding the acceptable staleness of data, responses provide the
|
|
`X-Consul-LastContact` header containing the time in milliseconds that a server
|
|
was last contacted by the leader node. The `X-Consul-KnownLeader` header also
|
|
indicates if there is a known leader. These can be used by clients to gauge the
|
|
staleness of a result and take appropriate action.
|
|
|
|
### Visibility into Consistency Mode Used
|
|
|
|
The Consul DNS and HTTP API interfaces allow setting the consistency mode
|
|
at different levels (interface and per-request).
|
|
Advanced settings for the [DNS](#limiting-staleness-advanced-usage) and
|
|
[HTTP API](#changing-the-default-consistency-mode-advanced-usage) interfaces can even
|
|
cause the consistency mode to change from `stale` to `default` if results are sufficiently stale.
|
|
|
|
To understand the consistency mode used for a particular HTTP API request,
|
|
check the `X-Consul-Effective-Consistency` response header.
|
|
The value of the header matches the
|
|
[name of the query parameter associated with the consistency mode](#overriding-a-request-s-consistency-mode).
|
|
In the following example, the header shows that the `default` consistency mode was used.
|
|
|
|
```shell-session
|
|
$ curl --include $CONSUL_HTTP_ADDR/v1/catalog/services
|
|
|
|
HTTP/1.1 200 OK
|
|
Content-Type: application/json
|
|
...
|
|
X-Consul-Effective-Consistency: leader
|
|
X-Consul-KnownLeader: true
|
|
X-Consul-LastContact: 0
|
|
|
|
{
|
|
"redis": [],
|
|
"postgresql": ["primary", "secondary"]
|
|
}
|
|
```
|
|
|
|
The DNS interface does not support viewing the consistency mode used for a particular query.
|
|
|
|
### Relationship to Request Caching
|
|
|
|
Note that some HTTP API endpoints support a `cached` parameter which has some of the same
|
|
semantics as `stale` consistency mode but different trade offs. This behavior is described in
|
|
[agent caching feature documentation](/api-docs/features/caching)
|
|
|
|
<!-- Common links references -->
|
|
[`dns_config.allow_stale`]: /docs/agent/options#allow_stale)
|
|
[`dns_config.max_stale`]: /docs/agent/options#max_stale
|
|
[`discovery_max_stale`]: /docs/agent/options#discovery_max_stale |