From e93e7d015258c9b963e9013fd963e25c04be037f Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Mon, 3 May 2021 17:27:25 -0400 Subject: [PATCH 1/5] docs: Add streaming to api features --- .../content/api-docs/features/streaming.mdx | 43 +++++++++++++++++++ website/data/api-docs-nav-data.json | 4 ++ 2 files changed, 47 insertions(+) create mode 100644 website/content/api-docs/features/streaming.mdx diff --git a/website/content/api-docs/features/streaming.mdx b/website/content/api-docs/features/streaming.mdx new file mode 100644 index 0000000000..8a2c912d0f --- /dev/null +++ b/website/content/api-docs/features/streaming.mdx @@ -0,0 +1,43 @@ +--- +layout: api +page_title: Streaming +description: |- + Some API endpoints support a feature known as "streaming". Change events + are published to a stream topic, allowing clients to build a localized + view, and removing the need to send full state-of-the-world responses + every time data changes. +--- + +# Streaming + +A few endpoints in Consul support a feature known as "streaming". With streaming +the Consul servers publish change events to a topic. Clients can subscribe to a +topic to receive these change events. The clients use the change events to build +a local "materialized view". Clients are then able to handle requests using that view. +Streaming improves on [blocking queries][1] by significantly reducing the quantity +of data sent from servers to clients, while still allowing clients to handle +read requests. + +Streaming, first introduced in Consul 1.10, is a replacement for [blocking queries][1], +and is only supported by a small number of endpoints. If streaming is supported by an +endpoint, it will be used when either the `index` or `cached` query parameters are set. + +Endpoints that support streaming return an HTTP header named +`X-Consul-Index`. This is a unique identifier representing the current state of +the requested resource. + +On subsequent requests for this resource, the client can set the `index` query +string parameter to the value of `X-Consul-Index`, indicating that the client +wishes to wait for any changes subsequent to that index. + +When `index` is set, the HTTP request will "hang" until a change in the system +occurs, or the maximum timeout is reached. A critical note is that the return of +a blocking request is **no guarantee** of a change. It is possible that the +timeout was reached or that there was an idempotent write that does not affect +the result of the query. + +Note that streaming does not support all query parameters used by blocking queries, and +that some response headers (ex: `X-Consul-LastContact` and `X-Consul-KnownLeader`) are not +set because the data is not available. + +[1]: /api-docs/features/blocking diff --git a/website/data/api-docs-nav-data.json b/website/data/api-docs-nav-data.json index 4f1688f561..2cf6017b9c 100644 --- a/website/data/api-docs-nav-data.json +++ b/website/data/api-docs-nav-data.json @@ -14,6 +14,10 @@ "title": "Blocking Queries", "path": "features/blocking" }, + { + "title": "Streaming", + "path": "features/streaming" + }, { "title": "Filtering", "path": "features/filtering" From 61423fbd28cda032f6ff6e02544124c897193119 Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Mon, 3 May 2021 17:40:07 -0400 Subject: [PATCH 2/5] Document streaming on service health endpoint --- website/content/api-docs/health.mdx | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/website/content/api-docs/health.mdx b/website/content/api-docs/health.mdx index 220244dc63..0688dc2252 100644 --- a/website/content/api-docs/health.mdx +++ b/website/content/api-docs/health.mdx @@ -208,14 +208,15 @@ incorporating the use of health checks. | `GET` | `/health/service/:service` | `application/json` | The table below shows this endpoint's support for +[streaming](/api/features/streaming), [blocking queries](/api/features/blocking), [consistency modes](/api/features/consistency), [agent caching](/api/features/caching), and [required ACLs](/api#authentication). -| Blocking Queries | Consistency Modes | Agent Caching | ACL Required | -| ---------------- | ----------------- | -------------------- | ------------------------ | -| `YES` | `all` | `background refresh` | `node:read,service:read` | +| Streaming | Blocking Queries | Consistency Modes | Agent Caching | ACL Required | +| ----------| ---------------- | ----------------- | -------------------- | ------------------------ | +| `partial` | `YES` | `all` | `background refresh` | `node:read,service:read` | ### Parameters @@ -425,6 +426,11 @@ gateway](/docs/connect/ingress-gateway) for a service in a given datacenter. Parameters and response format are the same as [`/health/service/:service`](/api/health#list-nodes-for-service). +**Note** that unlike `/health/connect/:service` and `/health/service/:service` this +endpoint does not support [streaming](/api/features/streaming), and will use +[blocking queries](/api/features/blocking). + + ## List Checks in State This endpoint returns the checks in the state provided on the path. From c19d5d831b3c52f4a7e7b9034e3b488e7af7feb5 Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Mon, 3 May 2021 17:43:26 -0400 Subject: [PATCH 3/5] docs: try to improve health api doc terminology --- website/content/api-docs/health.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/website/content/api-docs/health.mdx b/website/content/api-docs/health.mdx index 0688dc2252..26943f06a3 100644 --- a/website/content/api-docs/health.mdx +++ b/website/content/api-docs/health.mdx @@ -197,9 +197,9 @@ the following selectors and filter operations being supported: | `ServiceTags` | In, Not In, Is Empty, Is Not Empty | | `Status` | Equal, Not Equal, In, Not In, Matches, Not Matches | -## List Nodes for Service +## List Service Instances for Service ((list-nodes-for-service)) -This endpoint returns the nodes providing the service indicated on the path. +This endpoint returns the service instances providing the service indicated on the path. Users can also build in support for dynamic load balancing and other features by incorporating the use of health checks. @@ -397,9 +397,9 @@ following selectors and filter operations being supported: | `Service.Weights.Passing` | Equal, Not Equal | | `Service.Weights.Warning` | Equal, Not Equal | -## List Nodes for Connect-capable Service +## List Service Instances for Connect-enabled Service -This endpoint returns the nodes providing a +This endpoint returns the service instances providing a [Connect-capable](/docs/connect) service in a given datacenter. This will include both proxies and native integrations. A service may register both Connect-capable and incapable services at the same time, @@ -412,11 +412,11 @@ so this endpoint may be used to filter only the Connect-capable endpoints. Parameters and response format are the same as [`/health/service/:service`](/api/health#list-nodes-for-service). -## List Nodes for Ingress Gateways Associated to a Service +## List Service Instances for Ingress Gateways Associated with Service -> **1.8.0+:** This API is available in Consul versions 1.8.0 and later. -This endpoint returns the nodes providing a [ingress +This endpoint returns the service instances providing a [ingress gateway](/docs/connect/ingress-gateway) for a service in a given datacenter. | Method | Path | Produces | From c46f4391f0fa4813febd1f03a3fcdd6b2b02501d Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Tue, 8 Jun 2021 13:13:16 -0400 Subject: [PATCH 4/5] fixup! docs: Add streaming to api features --- website/data/api-docs-nav-data.json | 4 ---- 1 file changed, 4 deletions(-) diff --git a/website/data/api-docs-nav-data.json b/website/data/api-docs-nav-data.json index 2cf6017b9c..4f1688f561 100644 --- a/website/data/api-docs-nav-data.json +++ b/website/data/api-docs-nav-data.json @@ -14,10 +14,6 @@ "title": "Blocking Queries", "path": "features/blocking" }, - { - "title": "Streaming", - "path": "features/streaming" - }, { "title": "Filtering", "path": "features/filtering" From 8b9ec040c3c2a4d9f35f60cbf22218710a1f930e Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Tue, 8 Jun 2021 13:48:56 -0400 Subject: [PATCH 5/5] docs: move streaming docs to blocking query page --- .../content/api-docs/features/blocking.mdx | 18 ++++++++ .../content/api-docs/features/streaming.mdx | 43 ------------------- website/content/api-docs/health.mdx | 21 +++++---- 3 files changed, 30 insertions(+), 52 deletions(-) delete mode 100644 website/content/api-docs/features/streaming.mdx diff --git a/website/content/api-docs/features/blocking.mdx b/website/content/api-docs/features/blocking.mdx index 093bd1e545..2b7a9ae860 100644 --- a/website/content/api-docs/features/blocking.mdx +++ b/website/content/api-docs/features/blocking.mdx @@ -82,6 +82,24 @@ cases that must be handled correctly. [token bucket](https://en.wikipedia.org/wiki/Token_bucket) with burst of 2 is a simple way to achieve this. +## Streaming backend + +The streaming backend, first introduced in Consul 1.10, is a replacement for the long +polling backend. If streaming is supported by an endpoint, it will be used when either +the `index` or `cached` query parameters are set. + +When streaming is used, the Consul servers publish change events to a topic. Clients +subscribe to a topic to receive these change events. The clients use the change events to build +a local "materialized view". Clients are then able to handle requests using that view. +Streaming improves on the traditional long polling between clients and servers +by significantly reducing the quantity of data sent between them, while still allowing +clients to handle read requests. + +While streaming is a significant optimization over long polling, it will not populate the +`X-Consul-LastContact` or `X-Consul-KnownLeader` response headers, because the required +data is not available to the client. + + ## Hash-based Blocking Queries A limited number of agent endpoints also support blocking however because the diff --git a/website/content/api-docs/features/streaming.mdx b/website/content/api-docs/features/streaming.mdx deleted file mode 100644 index 8a2c912d0f..0000000000 --- a/website/content/api-docs/features/streaming.mdx +++ /dev/null @@ -1,43 +0,0 @@ ---- -layout: api -page_title: Streaming -description: |- - Some API endpoints support a feature known as "streaming". Change events - are published to a stream topic, allowing clients to build a localized - view, and removing the need to send full state-of-the-world responses - every time data changes. ---- - -# Streaming - -A few endpoints in Consul support a feature known as "streaming". With streaming -the Consul servers publish change events to a topic. Clients can subscribe to a -topic to receive these change events. The clients use the change events to build -a local "materialized view". Clients are then able to handle requests using that view. -Streaming improves on [blocking queries][1] by significantly reducing the quantity -of data sent from servers to clients, while still allowing clients to handle -read requests. - -Streaming, first introduced in Consul 1.10, is a replacement for [blocking queries][1], -and is only supported by a small number of endpoints. If streaming is supported by an -endpoint, it will be used when either the `index` or `cached` query parameters are set. - -Endpoints that support streaming return an HTTP header named -`X-Consul-Index`. This is a unique identifier representing the current state of -the requested resource. - -On subsequent requests for this resource, the client can set the `index` query -string parameter to the value of `X-Consul-Index`, indicating that the client -wishes to wait for any changes subsequent to that index. - -When `index` is set, the HTTP request will "hang" until a change in the system -occurs, or the maximum timeout is reached. A critical note is that the return of -a blocking request is **no guarantee** of a change. It is possible that the -timeout was reached or that there was an idempotent write that does not affect -the result of the query. - -Note that streaming does not support all query parameters used by blocking queries, and -that some response headers (ex: `X-Consul-LastContact` and `X-Consul-KnownLeader`) are not -set because the data is not available. - -[1]: /api-docs/features/blocking diff --git a/website/content/api-docs/health.mdx b/website/content/api-docs/health.mdx index 26943f06a3..77721a7cbc 100644 --- a/website/content/api-docs/health.mdx +++ b/website/content/api-docs/health.mdx @@ -197,7 +197,7 @@ the following selectors and filter operations being supported: | `ServiceTags` | In, Not In, Is Empty, Is Not Empty | | `Status` | Equal, Not Equal, In, Not In, Matches, Not Matches | -## List Service Instances for Service ((list-nodes-for-service)) +## List Service Instances for Service ((#list-nodes-for-service)) This endpoint returns the service instances providing the service indicated on the path. Users can also build in support for dynamic load balancing and other features by @@ -208,15 +208,19 @@ incorporating the use of health checks. | `GET` | `/health/service/:service` | `application/json` | The table below shows this endpoint's support for -[streaming](/api/features/streaming), [blocking queries](/api/features/blocking), [consistency modes](/api/features/consistency), [agent caching](/api/features/caching), and [required ACLs](/api#authentication). -| Streaming | Blocking Queries | Consistency Modes | Agent Caching | ACL Required | -| ----------| ---------------- | ----------------- | -------------------- | ------------------------ | -| `partial` | `YES` | `all` | `background refresh` | `node:read,service:read` | +| Blocking Queries | Consistency Modes | Agent Caching | ACL Required | +| -------------------- | ----------------- | -------------------- | ------------------------ | +| `YES` 1 | `all` | `background refresh` | `node:read,service:read` | + +

+ 1some query parameters will use the + streaming backend +

### Parameters @@ -412,11 +416,11 @@ so this endpoint may be used to filter only the Connect-capable endpoints. Parameters and response format are the same as [`/health/service/:service`](/api/health#list-nodes-for-service). -## List Service Instances for Ingress Gateways Associated with Service +## List Service Instances for Ingress Gateways Associated with a Service -> **1.8.0+:** This API is available in Consul versions 1.8.0 and later. -This endpoint returns the service instances providing a [ingress +This endpoint returns the service instances providing an [ingress gateway](/docs/connect/ingress-gateway) for a service in a given datacenter. | Method | Path | Produces | @@ -427,8 +431,7 @@ Parameters and response format are the same as [`/health/service/:service`](/api/health#list-nodes-for-service). **Note** that unlike `/health/connect/:service` and `/health/service/:service` this -endpoint does not support [streaming](/api/features/streaming), and will use -[blocking queries](/api/features/blocking). +endpoint does not support the [streaming backend](/api/features/blocking#streaming-backend). ## List Checks in State