Update docs to include multiple tag support (#4797)

* Update docs to include multiple tag support

* Sort tags before using them in metrics

This addresses the potential proliferation of metrics if a query of
"?tag=foo&tag=bar" is treated differently than "?tag=bar&tag=foo".
Now, tags are always sorted before being recorded, making these two
emit the same metric.

* Add caveat about multiple tags returned by the metrics endpoint

agent/consul/catalog_endpoint.go

@@ -2,6 +2,7 @@ package consul
 
 import (
 	"fmt"
+	"sort"
 	"time"
 
 	"github.com/armon/go-metrics"
@@ -335,6 +336,10 @@ func (c *Catalog) ServiceNodes(args *structs.ServiceSpecificRequest, reply *stru
 				[]metrics.Label{{Name: "service", Value: args.ServiceName}, {Name: "tag", Value: args.ServiceTag}})
 		}
 		if len(args.ServiceTags) > 0 {
+			// Sort tags so that the metric is the same even if the request
+			// tags are in a different order
+			sort.Strings(args.ServiceTags)
+
 			// Build metric labels
 			labels := []metrics.Label{{Name: "service", Value: args.ServiceName}}
 			for _, tag := range args.ServiceTags {

agent/consul/health_endpoint.go

@@ -2,6 +2,7 @@ package consul
 
 import (
 	"fmt"
+	"sort"
 
 	"github.com/armon/go-metrics"
 	"github.com/hashicorp/consul/agent/consul/state"
@@ -171,6 +172,10 @@ func (h *Health) ServiceNodes(args *structs.ServiceSpecificRequest, reply *struc
 				[]metrics.Label{{Name: "service", Value: args.ServiceName}, {Name: "tag", Value: args.ServiceTag}})
 		}
 		if len(args.ServiceTags) > 0 {
+			// Sort tags so that the metric is the same even if the request
+			// tags are in a different order
+			sort.Strings(args.ServiceTags)
+
 			labels := []metrics.Label{{Name: "service", Value: args.ServiceName}}
 			for _, tag := range args.ServiceTags {
 				labels = append(labels, metrics.Label{Name: "tag", Value: tag})

website/source/api/agent.html.md

@@ -246,6 +246,11 @@ In order to enable [Prometheus](https://prometheus.io/) support, you need to use
 configuration directive
 [`prometheus_retention_time`](/docs/agent/options.html#telemetry-prometheus_retention_time).
 
+Note: If your metric includes labels that use the same key name multiple times 
+(i.e. tag=tag2 and tag=tag1), only the sorted last value (tag=tag2) will be visible on 
+this endpoint due to a display issue. The complete label set is correctly applied and 
+passed to external metrics providers even though it is not visible through this endpoint.
+
 | Method | Path                               | Produces                                   |
 | ------ | ---------------------------------- | ------------------------------------------ |
 | `GET`  | `/agent/metrics`                   | `application/json`                         |

website/source/api/catalog.html.md

@@ -399,7 +399,9 @@ The table below shows this endpoint's support for
   the datacenter of the agent being queried. This is specified as part of the
   URL as a query parameter.
 
-- `tag` `(string: "")` - Specifies the tag to filter on.
+- `tag` `(string: "")` - Specifies the tag to filter on. This is specified as part of 
+  the URL as a query parameter. Can be used multiple times for additional filtering,
+  returning only the results that include all of the tag values provided.
 
 - `near` `(string: "")` - Specifies a node name to sort the node list in
   ascending order based on the estimated round trip time from that node. Passing

website/source/api/health.html.md

@@ -178,7 +178,9 @@ The table below shows this endpoint's support for
   part of the URL as a query parameter.
 
 - `tag` `(string: "")` - Specifies the tag to filter the list. This is
-  specifies as part of the URL as a query parameter.
+  specified as part of the URL as a query parameter. Can be used multiple times 
+  for additional filtering, returning only the results that include all of the tag 
+  values provided.
 
 - `node-meta` `(string: "")` - Specifies a desired node metadata key/value pair
   of the form `key:value`. This parameter can be specified multiple times, and

website/source/docs/agent/telemetry.html.md

@@ -933,6 +933,12 @@ These metrics give insight into the health of the cluster as a whole.
     <td>queries</td>
     <td>counter</td>
   </tr>
+  <tr>
+    <td>`consul.catalog.service.query-tags.<service>.<tags>`</td>
+    <td>This increments for each catalog query for the given service with the given tags.</td>
+    <td>queries</td>
+    <td>counter</td>
+  </tr>
   <tr>
     <td>`consul.catalog.service.not-found.<service>`</td>
     <td>This increments for each catalog query where the given service could not be found.</td>
@@ -951,6 +957,12 @@ These metrics give insight into the health of the cluster as a whole.
     <td>queries</td>
     <td>counter</td>
   </tr>
+  <tr>
+    <td>`consul.health.service.query-tags.<service>.<tags>`</td>
+    <td>This increments for each health query for the given service with the given tags.</td>
+    <td>queries</td>
+    <td>counter</td>
+  </tr>
   <tr>
     <td>`consul.health.service.not-found.<service>`</td>
     <td>This increments for each health query where the given service could not be found.</td>