From 1f5398e17c9465164746e71e6d852562daa156b2 Mon Sep 17 00:00:00 2001 From: Mitchell Hashimoto Date: Wed, 6 Jun 2018 15:10:52 -0700 Subject: [PATCH] website: document multi-DC, caching, clarify prepared queries and multi-DC --- website/source/docs/connect/index.html.md | 51 +++++++++++++++++++++ website/source/docs/connect/proxies.html.md | 6 +++ 2 files changed, 57 insertions(+) diff --git a/website/source/docs/connect/index.html.md b/website/source/docs/connect/index.html.md index 3b459688b2..ff79b48c0c 100644 --- a/website/source/docs/connect/index.html.md +++ b/website/source/docs/connect/index.html.md @@ -64,3 +64,54 @@ APIs are all made to the local Consul agent over a loopback interface, and all local caching, background updating, and support blocking queries. As a result, most API calls operate on purely local in-memory data and can respond in microseconds. + +## Agent Caching and Performance + +To enable microsecond-speed responses on +[agent Connect API endpoints](/api/agent/connect.html), the Consul agent +locally caches most Connect-related data and sets up background +[blocking queries](/api/index.html#blocking-queries) against the server +to update the cache in the background. This allows most API calls such +as retrieving certificates or authorizing connections to use in-memory +data and respond very quickly. + +All data cached locally by the agent is populated on demand. Therefore, +if Connect is not used at all, the cache does not store any data. On first +request, the data is loaded from the server and cached. The set of data cached +is: public CA root certificates, leaf certificates, and intentions. For +leaf certificates and intentions, only data related to the service requested +is cached, not the full set of data. + +Further, the cache is partitioned by ACL token and datacenters. This is done +to minimize the complexity of the cache and prevent bugs where an ACL token +may see data it shouldn't from the cache. This results in higher memory usage +for cached data since it is duplicated per ACL token, but with the benefit +of simplicity and security. + +With Connect enabled, you'll likely see increased memory usage by the +local Consul agent. The total memory is dependent on the number of intentions +related to the services registered with the agent accepting Connect-based +connections. The other data (leaf certificates and public CA certificates) +is a relatively fixed size per service. In most cases, the overhead per +service should be relatively small: single digit kilobytes at most. + +The cache does not evict entries due to memory pressure. If memory capacity +is reached, the process will attempt to swap. If swap is disabled, the Consul +agent may begin failing and eventually crash. Cache entries do have TTLs +associated with them and will evict their entries if they're not used. Given +a long period of inactivity (3 days by default), the cache will empty itself. + +## Multi-Datacenter + +Connect currently only works for service-to-service connections wtihin a +single Consul datacenter. Connect may be enabled on multiple Consul datacenters, +but only services within the same datacenters can establish Connect-based +connections. +CA configurations and intentions are both local to their respective datacenters; +they are not replicated across datacenters. + +Multi-datacenter support for Connect is under development and will be +released as a feature of Consul Enterprise in late 2018. This feature will +facilitate intention replication, datacenter constraints on intentions, +CA state replication, multi-datacenter certificate rotations, and more. + diff --git a/website/source/docs/connect/proxies.html.md b/website/source/docs/connect/proxies.html.md index b588c66656..ed49225c2c 100644 --- a/website/source/docs/connect/proxies.html.md +++ b/website/source/docs/connect/proxies.html.md @@ -133,6 +133,12 @@ service. } ``` +-> **Note:** Connect does not currently support cross-datacenter +service communication. Therefore, prepared queries with Connect should +only be used to discover services within a single datacenter. See +[Multi-Datacenter Connect](/docs/connect/index.html#multi-datacenter) for +more information. + ### Dynamic Upstreams If an application requires dynamic dependencies that are only available