2014-02-20 00:58:15 +00:00
---
2020-04-07 18:55:19 +00:00
layout: docs
page_title: Consul Architecture
2020-04-13 18:40:26 +00:00
sidebar_title: Architecture
2020-04-07 18:55:19 +00:00
description: >-
Consul is a complex system that has many different moving parts. To help users
and developers of Consul form a mental model of how it works, this page
documents the system architecture.
2014-02-20 00:58:15 +00:00
---
# Consul Architecture
Consul is a complex system that has many different moving parts. To help
users and developers of Consul form a mental model of how it works, this
page documents the system architecture.
2020-04-06 20:27:35 +00:00
-> Before describing the architecture, we recommend reading the
2020-04-09 23:46:54 +00:00
[glossary](/docs/glossary) of terms to help
2019-06-24 21:19:12 +00:00
clarify what is being discussed.
2014-02-20 00:58:15 +00:00
2020-09-10 17:32:06 +00:00
The architecture concepts in this document can be used with the [Reference Architecture guide](https://learn.hashicorp.com/tutorials/consul/reference-architecture?utm_source=consul.io&utm_medium=docs) when deploying Consul in production.
2014-02-20 00:58:15 +00:00
## 10,000 foot view
From a 10,000 foot altitude the architecture of Consul looks like this:
2020-04-07 23:56:08 +00:00
[](/img/consul-arch.png)
2014-02-20 00:58:15 +00:00
2015-03-30 22:07:58 +00:00
Let's break down this image and describe each piece. First of all, we can see
that there are two datacenters, labeled "one" and "two". Consul has first
2020-09-10 17:32:06 +00:00
class support for [multiple datacenters](https://learn.hashicorp.com/tutorials/consul/federarion-gossip-wan) and
2015-03-30 22:07:58 +00:00
expects this to be the common case.
2014-02-20 00:58:15 +00:00
2015-03-30 22:07:58 +00:00
Within each datacenter, we have a mixture of clients and servers. It is expected
2020-09-22 21:57:46 +00:00
that there will be between three to five servers. This strikes a balance between
2014-02-20 00:58:15 +00:00
availability in the case of failure and performance, as consensus gets progressively
slower as more machines are added. However, there is no limit to the number of clients,
and they can easily scale into the thousands or tens of thousands.
2020-04-09 23:46:54 +00:00
All the agents that are in a datacenter participate in a [gossip protocol](/docs/internals/gossip).
2019-06-24 17:25:47 +00:00
This means there is a gossip pool that contains all the agents for a given datacenter. This serves
2014-11-26 12:31:38 +00:00
a few purposes: first, there is no need to configure clients with the addresses of servers;
2019-06-24 17:25:47 +00:00
discovery is done automatically. Second, the work of detecting agent failures
2014-11-26 12:31:38 +00:00
is not placed on the servers but is distributed. This makes failure detection much more
2019-08-01 18:21:37 +00:00
scalable than naive heartbeating schemes. It also provides failure detection for the nodes; if the agent is not reachable, then the node may have experienced a failure. Thirdly, it is used as a messaging layer to notify
2014-02-20 00:58:15 +00:00
when important events such as leader election take place.
The servers in each datacenter are all part of a single Raft peer set. This means that
2015-03-30 22:07:58 +00:00
they work together to elect a single leader, a selected server which has extra duties. The leader
is responsible for processing all queries and transactions. Transactions must also be replicated to
2020-04-09 23:46:54 +00:00
all peers as part of the [consensus protocol](/docs/internals/consensus). Because of this
2015-03-30 22:07:58 +00:00
requirement, when a non-leader server receives an RPC request, it forwards it to the cluster leader.
2014-02-20 00:58:15 +00:00
2019-06-24 17:25:47 +00:00
The server agents also operate as part of a WAN gossip pool. This pool is different from the LAN pool
2015-03-30 22:07:58 +00:00
as it is optimized for the higher latency of the internet and is expected to contain only
2019-06-24 17:25:47 +00:00
other Consul server agents. The purpose of this pool is to allow datacenters to discover each
2015-03-30 22:07:58 +00:00
other in a low-touch manner. Bringing a new datacenter online is as easy as joining the existing
2018-01-04 21:44:07 +00:00
WAN gossip pool. Because the servers are all operating in this pool, it also enables cross-datacenter
2015-03-30 22:07:58 +00:00
requests. When a server receives a request for a different datacenter, it forwards it to a random
server in the correct datacenter. That server may then forward to the local leader.
2014-02-20 00:58:15 +00:00
2014-02-20 01:05:57 +00:00
This results in a very low coupling between datacenters, but because of failure detection,
2014-04-09 18:06:27 +00:00
connection caching and multiplexing, cross-datacenter requests are relatively fast and reliable.
2014-02-20 00:58:15 +00:00
2017-08-04 23:14:39 +00:00
In general, data is not replicated between different Consul datacenters. When a
request is made for a resource in another datacenter, the local Consul servers forward
an RPC request to the remote Consul servers for that resource and return the results.
If the remote datacenter is not available, then those resources will also not be
available, but that won't otherwise affect the local datacenter. There are some special
situations where a limited subset of data can be replicated, such as with Consul's built-in
2020-09-10 17:32:06 +00:00
[ACL replication](https://learn.hashicorp.com/tutorials/consul/access-control-replication-multiple-datacenters) capability, or
2017-08-04 23:14:39 +00:00
external tools like [consul-replicate](https://github.com/hashicorp/consul-replicate).
2018-09-06 10:34:28 +00:00
In some places, client agents may cache data from the servers to make it
available locally for performance and reliability. Examples include Connect
certificates and intentions which allow the client agent to make local decisions
about inbound connection requests without a round trip to the servers. Some API
endpoints also support optional result caching. This helps reliability because
the local agent can continue to respond to some queries like service-discovery
or Connect authorization from cache even if the connection to the servers is
disrupted or the servers are temporarily unavailable.
2014-02-20 00:58:15 +00:00
## Getting in depth
2014-11-26 12:31:38 +00:00
At this point we've covered the high level architecture of Consul, but there are many
2020-04-09 23:46:54 +00:00
more details for each of the subsystems. The [consensus protocol](/docs/internals/consensus) is
documented in detail as is the [gossip protocol](/docs/internals/gossip). The [documentation](/docs/internals/security)
2014-04-16 04:01:12 +00:00
for the security model and protocols used are also available.
2014-02-20 00:58:15 +00:00
2015-03-30 22:07:58 +00:00
For other details, either consult the code, ask in IRC, or reach out to the mailing list.
