Commit Graph

2705 Commits

Author SHA1 Message Date
Kenia 0ee745c899 Create consul version metric with version label (#9350)
* create consul version metric with version label

* agent/agent.go: add pre-release Version as well as label

Co-Authored-By: Radha13 <kumari.radha3@gmail.com>

* verion and pre-release version labels.

* hyphen/- breaks prometheus

* Add Prometheus gauge defintion for version metric

* Add new metric to telemetry docs

Co-authored-by: Radha Kumari <kumari.radha3@gmail.com>
Co-authored-by: Aestek <thib.gilles@gmail.com>
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2020-12-09 14:17:51 +00:00
Kyle Havlovitz 38bbf32a9c Merge pull request #9318 from hashicorp/ca-update-followup
connect: Fix issue with updating config in secondary
2020-12-02 20:18:26 +00:00
Kyle Havlovitz ff93919034 Merge pull request #9009 from hashicorp/update-secondary-ca
connect: Fix an issue with updating CA config in a secondary datacenter
2020-11-30 22:50:26 +00:00
Daniel Nephin 60d7f30169 Merge pull request #9284 from hashicorp/dnephin/agent-service-register
local: mark service as InSync when added to local agent state
2020-11-27 20:50:53 +00:00
Hans Hasselberg b3a0b8edd9 fix serf_wan documentation (#9289)
WAN config is different than LAN config, source of truth is
f72d2042a8/config.go (L315-L326)
and now the docs are correct.
2020-11-27 19:51:04 +00:00
Daniel Nephin d230cea541 Merge pull request #9247 from pierresouchay/streaming_predictible_order_for_health
[Streaming] Predictable order for results of /health/service/:serviceName to mimic memdb
2020-11-25 20:55:00 +00:00
Mike Morris 3ee6d1c14f
Merge branch 'release/1.9.x' into release/1.9.0 2020-11-24 14:50:39 -05:00
Mike Morris dbb1249f13 Merge branch 'stable-website' into release/1.9.0 2020-11-24 14:44:53 -05:00
hashicorp-ci 79bb27a363
update bindata_assetfs.go 2020-11-24 19:05:48 +00:00
R.B. Boyer 7467ffbff3 server: fix panic when deleting a non existent intention (#9254)
* server: fix panic when deleting a non existent intention

* add changelog

* Always return an error when deleting non-existent ixn

Co-authored-by: freddygv <gh@freddygv.xyz>
2020-11-24 18:44:58 +00:00
R.B. Boyer 3c7cf0216d server: fix panic when deleting a non existent intention (#9254)
* server: fix panic when deleting a non existent intention

* add changelog

* Always return an error when deleting non-existent ixn

Co-authored-by: freddygv <gh@freddygv.xyz>
2020-11-24 13:44:45 -05:00
Kit Patella 727780140e Merge pull request #9261 from hashicorp/telemetry/fix-missing-and-stale-docs-2
Telemetry/fix missing and stale docs
2020-11-23 21:34:59 +00:00
Kit Patella 146466a708 Merge pull request #9261 from hashicorp/telemetry/fix-missing-and-stale-docs-2
Telemetry/fix missing and stale docs
2020-11-23 21:34:55 +00:00
Freddy ff5215d882 Require operator:write to get Connect CA config (#9240)
A vulnerability was identified in Consul and Consul Enterprise (“Consul”) such that operators with `operator:read` ACL permissions are able to read the Consul Connect CA configuration when explicitly configured with the `/v1/connect/ca/configuration` endpoint, including the private key. This allows the user to effectively privilege escalate by enabling the ability to mint certificates for any Consul Connect services. This would potentially allow them to masquerade (receive/send traffic) as any service in the mesh.

--

This PR increases the permissions required to read the Connect CA's private key when it was configured via the `/connect/ca/configuration` endpoint. They are now `operator:write`.
2020-11-23 06:27:20 -07:00
Kit Patella fe6ef7e414 Merge pull request #9245 from hashicorp/telemetry/fix-missing-and-stale-docs
Telemetry/fix missing and stale docs
2020-11-20 20:55:51 +00:00
Kit Patella 6e607d7cd3 Merge pull request #9245 from hashicorp/telemetry/fix-missing-and-stale-docs
Telemetry/fix missing and stale docs
2020-11-20 20:55:45 +00:00
Freddy 4e44341d36 Require operator:write to get Connect CA config (#9240)
A vulnerability was identified in Consul and Consul Enterprise (“Consul”) such that operators with `operator:read` ACL permissions are able to read the Consul Connect CA configuration when explicitly configured with the `/v1/connect/ca/configuration` endpoint, including the private key. This allows the user to effectively privilege escalate by enabling the ability to mint certificates for any Consul Connect services. This would potentially allow them to masquerade (receive/send traffic) as any service in the mesh.

--

This PR increases the permissions required to read the Connect CA's private key when it was configured via the `/connect/ca/configuration` endpoint. They are now `operator:write`.
2020-11-19 16:50:17 -07:00
R.B. Boyer 140c220131
[1.9.0] command: when generating envoy bootstrap configs use the datacenter returned from the agent services endpoint (#9230)
Manual backport of #9229 into 1.9.0 branch

Fixes #9215
2020-11-19 15:33:41 -06:00
R.B. Boyer 32f6d17e5d command: when generating envoy bootstrap configs use the datacenter returned from the agent services endpoint (#9229)
Fixes #9215
2020-11-19 21:28:09 +00:00
Freddy 5137e4501d Require operator:write to get Connect CA config (#9240)
A vulnerability was identified in Consul and Consul Enterprise (“Consul”) such that operators with `operator:read` ACL permissions are able to read the Consul Connect CA configuration when explicitly configured with the `/v1/connect/ca/configuration` endpoint, including the private key. This allows the user to effectively privilege escalate by enabling the ability to mint certificates for any Consul Connect services. This would potentially allow them to masquerade (receive/send traffic) as any service in the mesh.

--

This PR increases the permissions required to read the Connect CA's private key when it was configured via the `/connect/ca/configuration` endpoint. They are now `operator:write`.
2020-11-19 17:15:17 +00:00
Daniel Nephin 6e5f062593 Merge pull request #9224 from hashicorp/dnephin/fix-multiple-http-listeners
agent: fix bug with multiple listeners
2020-11-18 21:53:27 +00:00
Daniel Nephin b2c5e2d059 Use freeport
To prevent other tests which already use freeport from flaking when port 0 steals their reserved port.
2020-11-18 16:07:00 -05:00
Daniel Nephin c6381b7e2b agent: fix bug with multiple listeners
Previously the listener was being passed to a closure in a loop without
capturing the loop variable. The result is only the last listener is
used, so the http/https servers only listen on one address.

This problem is fixed by capturing the variable by passing it into a
function.
2020-11-18 14:39:26 -05:00
Daniel Nephin 6a42641eb2 Merge pull request #9160 from hashicorp/dnephin/go-test-race-in-to-out-list
ci: change go-test-race package list to exclude list
2020-11-17 18:14:09 +00:00
hashicorp-ci 2dfe2401ec Merge branch 'release/1.9.0-rc1' into remote-x 2020-11-17 17:43:03 +00:00
hashicorp-ci 15ef28f57a
update bindata_assetfs.go 2020-11-17 16:28:08 +00:00
Matt Keeler dfaaa0b73a Refactor to call non-voting servers read replicas (#9191)
Co-authored-by: Kit Patella <kit@jepsen.io>
2020-11-17 15:54:38 +00:00
Freddy ef7ee6840a Add DC and NS support for Envoy metrics (#9207)
This PR updates the tags that we generate for Envoy stats.

Several of these come with breaking changes, since we can't keep two stats prefixes for a filter.
2020-11-16 19:55:18 -07:00
Kit Patella 88b013be99 Merge pull request #9198 from hashicorp/mkcp/telemetry/add-all-metric-definitions
Add metric definitions for all metrics known at Consul start
2020-11-16 16:26:16 -08:00
Kit Patella 82e7363b90 Merge pull request #9198 from hashicorp/mkcp/telemetry/add-all-metric-definitions
Add metric definitions for all metrics known at Consul start
2020-11-17 00:13:51 +00:00
Freddy 4d39305442 Add DC and NS support for Envoy metrics (#9207)
This PR updates the tags that we generate for Envoy stats.

Several of these come with breaking changes, since we can't keep two stats prefixes for a filter.
2020-11-16 23:37:50 +00:00
Matt Keeler dd857bfa37
Prevent panic if autopilot health is requested prior to leader establishment finishing. (#9204) 2020-11-16 17:14:56 -05:00
Matt Keeler e421da3b59 Prevent panic if autopilot health is requested prior to leader establishment finishing. (#9204) 2020-11-16 22:08:44 +00:00
Daniel Nephin 9b904de406 Merge pull request #9114 from hashicorp/dnephin/filtering-in-stream
stream: improve naming of Payload methods
2020-11-16 19:21:20 +00:00
R.B. Boyer 2747b5145a server: intentions CRUD requires connect to be enabled (#9194)
Fixes #9123
2020-11-13 22:19:47 +00:00
R.B. Boyer fee0c44ab2 server: remove config entry CAS in legacy intention API bridge code (#9151)
Change so line-item intention edits via the API are handled via the state store instead of via CAS operations.

Fixes #9143
2020-11-13 20:42:57 +00:00
R.B. Boyer a955705e5e server: skip deleted and deleting namespaces when migrating intentions to config entries (#9186) 2020-11-13 19:57:12 +00:00
Mike Morris 0ba0391bdd ci: update to Go 1.15.4 and alpine:3.12 (#9036)
* ci: stop building darwin/386 binaries

Go 1.15 drops support for 32-bit binaries on Darwin https://golang.org/doc/go1.15#darwin

* tls: ConnectionState::NegotiatedProtocolIsMutual is deprecated in Go 1.15, this value is always true

* correct error messages that changed slightly

* Completely regenerate some TLS test data

Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-11-13 18:03:37 +00:00
R.B. Boyer d69640a6e9 server: break up Intention.Apply monolithic method (#9007)
The Intention.Apply RPC is quite large, so this PR attempts to break it down into smaller functions and dissolves the pre-config-entry approach to the breakdown as it only confused things.
2020-11-13 15:16:34 +00:00
Daniel Nephin 83338d7f9a Merge pull request #9162 from hashicorp/dnephin/fix-grpc-metrics
grpc: fix metrics
2020-11-12 22:04:18 +00:00
R.B. Boyer f815014432 agent: return the default ACL policy to callers as a header (#9101)
Header is: X-Consul-Default-ACL-Policy=<allow|deny>

This is of particular utility when fetching matching intentions, as the
fallthrough for a request that doesn't match any intentions is to
enforce using the default acl policy.
2020-11-12 16:39:16 +00:00
Matt Keeler 1f4da2ae9d Add a CLI command for retrieving the autopilot configuration. (#9142) 2020-11-11 18:19:32 +00:00
Mike Morris 9c989fef4d
Merge pull request #9155 from hashicorp/release/1.9.0-beta3
merge: 1.9.0-beta3
2020-11-11 12:55:23 -05:00
Daniel Nephin 07bfd1d46e Merge pull request #8976 from joel0/wrap-eof
Wrap rpc error object
2020-11-11 16:51:42 +00:00
Matt Keeler e669899abf Add a paramter in state store methods to indicate whether a resource insertion is from a snapshot restoration (#9156)
The Catalog, Config Entry, KV and Session resources potentially re-validate the input as its coming in. We need to prevent snapshot restoration failures due to missing namespaces or namespaces that are being deleted in enterprise.
2020-11-11 16:22:11 +00:00
Daniel Nephin b3368b8ee7 Merge pull request #9149 from joel0/wrap-errors
Use error wrapping to preserve error type info
2020-11-10 23:27:42 +00:00
hashicorp-ci ab5265479f
update bindata_assetfs.go 2020-11-10 01:18:51 +00:00
Matt Keeler 8539565046 Merge pull request #9103 from hashicorp/feature/autopilot-mod
Switch to using the external autopilot module
2020-11-09 16:30:48 +00:00
Mike Morris 4f1d2a1c56 chore: upgrade to gopsutil/v3 (#9118)
* deps: update golang.org/x/sys

* deps: update imports to gopsutil/v3

* chore: make update-vendor
2020-11-07 01:49:01 +00:00
Mike Morris b721e1956a chore: update bindata_assetfs.go 2020-11-06 14:47:30 -05:00
R.B. Boyer fff77349ed Revert "Add namespace support for metrics (OSS) (#9117)" (#9124)
This reverts commit 06b3b017d3.
2020-11-06 16:57:56 +00:00
Freddy e86f58b163 Add namespace support for metrics (OSS) (#9117) 2020-11-05 18:30:37 -07:00
R.B. Boyer c464630615 agent: sanitize ui metrics proxy header values on agent/self endpoint (#9104) 2020-11-05 19:26:11 +00:00
Daniel Nephin 82a753285b Merge pull request #9073 from hashicorp/dnephin/backport-streaming-namespaces
streaming: backport namespace changes
2020-11-05 19:19:49 +00:00
Daniel Nephin f0beecad24 Merge pull request #9061 from hashicorp/dnephin/event-fields
stream: support filtering by namespace
2020-11-05 19:19:02 +00:00
R.B. Boyer c14439a146 agent: protect the ui metrics proxy endpoint behind ACLs (#9099)
This ensures the metrics proxy endpoint is ACL protected behind a
wildcard `service:read` and `node:read` set of rules. For Consul
Enterprise these will need to span all namespaces:

```
service_prefix "" { policy = "read" }
node_prefix ""    { policy = "read" }

namespace_prefix "" {
  service_prefix "" { policy = "read" }
  node_prefix ""    { policy = "read" }
}
```

This PR contains just the backend changes. The frontend changes to
actually pass the consul token header to the proxy through the JS plugin
will come in another PR.
2020-11-04 18:52:43 +00:00
Daniel Nephin b577dc19bf Merge pull request #9068 from hashicorp/restore-test-signature
restore prior signature of test helper so enterprise compiles
2020-11-02 23:16:21 +00:00
R.B. Boyer d6e7bf40f8 agent: introduce path allow list for requests going through the metrics proxy (#9059)
Added a new option `ui_config.metrics_proxy.path_allowlist`. This defaults to `["/api/v1/query", "/api/v1/query_range"]` when the metrics provider is set to `prometheus`.

Requests that do not use one of the allow-listed paths (via exact match) get a 403 Forbidden response instead.
2020-10-30 21:50:28 +00:00
R.B. Boyer 211051d92b state: ensure we unblock intentions queries upon the upgrade to config entries (#9062)
1. do a state store query to list intentions as the agent would do over in `agent/proxycfg` backing `agent/xds`
2. upgrade the database and do a fresh `service-intentions` config entry write
3. the blocking query inside of the agent cache in (1) doesn't notice (2)
2020-10-29 20:29:07 +00:00
Daniel Nephin 3fca80a52e Merge pull request #9025 from hashicorp/dnephin/streaming-options
streaming: Use a no-op event publisher if streaming is disabled
2020-10-29 19:31:08 +00:00
Daniel Nephin 7b9ee25956
Merge pull request #9026 from hashicorp/dnephin/streaming-without-cache-query-param
streaming: rename config and remove requirement for cache=1
2020-10-28 12:33:25 -04:00
Daniel Nephin 477d665309
Merge pull request #8618 from hashicorp/dnephin/remove-txn-readtxn
state: Use ReadTxn everywhere
2020-10-28 12:32:47 -04:00
Kyle Havlovitz 9f893307de
Merge pull request #9053 from hashicorp/vault-token-lookupself
connect: Use the lookup-self endpoint for Vault token
2020-10-27 14:34:03 -07:00
Daniel Nephin f0ac093fef agent/grpc: add connection count metrics
Gauge metrics are great for understanding the current state, but can somtimes hide problems
if there are many disconnect/reconnects.

This commit adds counter metrics for connections and streams to make it easier to see the
count of newly created connections and streams.
2020-10-27 16:49:49 -04:00
Daniel Nephin 5319ba02b0 agent/grpc: rename metrics
These new names should make it easier to add counter metics with similar prefixes
2020-10-27 16:49:49 -04:00
Daniel Nephin c82d6aa4ff
Merge pull request #8961 from hashicorp/dnephin/grpc-resolve-node-id
agent/grpc: fix some test flakes and handle duplicate server IDs in the pool
2020-10-27 16:47:37 -04:00
Daniel Nephin 74ac34e358
Merge pull request #8998 from hashicorp/dnephin/lib-ttlcache
lib/ttlcache: extract a new package from agent/cache
2020-10-27 16:43:10 -04:00
Daniel Nephin 0f81915495
Merge pull request #8987 from hashicorp/dnephin/stream-filter
streaming: apply filter to a single item
2020-10-27 16:39:43 -04:00
Daniel Nephin 8bcd5040c7 agent/grpc: Add an integration test for ClientPool with TLS
Also deregister the resolver.Builder in tests.
2020-10-27 16:34:18 -04:00
Daniel Nephin 8a785a351c agent/grpc: pass metrics to constructor
Instead of referencing a package var. This does not fix the flaky test, but it seems more correct.
2020-10-27 16:34:17 -04:00
Daniel Nephin d19657404f agent/grpc: fix a flaky test by performing more retries
Instead of using retry.Run, which appears to have problems in some cases where it does not
emit an error message, use a for loop.

Increase the number of attempts and remove any sleep, since this operation is not that expensive to do
in a tight loop
2020-10-27 16:34:17 -04:00
Daniel Nephin df405ac978 agent/grpc: remove misleading warnings from test output
Handle shutdown properly in tests so that the tests don't warn about using a closed connection.
2020-10-27 16:34:16 -04:00
Daniel Nephin e101aa8a74 agent/grpc: fix a flake in TestHandler_EmitsStats 2020-10-27 16:34:16 -04:00
Daniel Nephin 19da9c3a9b agent/grpc: use a separate channel for closing the Accept
Closing l.conns can lead to a race and a 'panic: send on closed chan' when a
connection is in the middle of being handled when the server is shutting down.

Found using '-race -count=800'
2020-10-27 16:34:15 -04:00
Daniel Nephin d8299670cc agent/grpc/resolver: namespace the server ID with the DC name
So that if two datacenters end up with overlapping serverIDs we don't send requests to the wrong server
2020-10-27 16:34:15 -04:00
Kyle Havlovitz f700a5707b connect: Use the lookup-self endpoint for Vault token 2020-10-27 13:03:45 -07:00
hashicorp-ci bea3d0fd96 auto-updated agent/uiserver/bindata_assetfs.go from commit f4208b5fb 2020-10-27 14:56:48 +00:00
hashicorp-ci 7a1538f747 auto-updated agent/uiserver/bindata_assetfs.go from commit 30da884d5 2020-10-27 14:31:16 +00:00
hashicorp-ci b6113eba09 auto-updated agent/uiserver/bindata_assetfs.go from commit ed6a2c150 2020-10-27 14:09:17 +00:00
hashicorp-ci 6c39fcecf3 auto-updated agent/uiserver/bindata_assetfs.go from commit 827e53694 2020-10-27 13:36:44 +00:00
hashicorp-ci 5460744a95 auto-updated agent/uiserver/bindata_assetfs.go from commit 52d7283cd 2020-10-26 19:56:11 +00:00
Paul Banks 52d7283cd6
UI metrics provider dc (#9001)
* Plumb Datacenter and Namespace to metrics provider in preparation for them being usable.

* Move metrics loader/status to a new component and show reason for being disabled.

* Remove stray console.log

* Rebuild AssetFS to resolve conflicts

* Yarn upgrade

* mend
2020-10-26 19:48:23 +00:00
hashicorp-ci f585db82cf auto-updated agent/uiserver/bindata_assetfs.go from commit c4f027fa0 2020-10-26 16:55:52 +00:00
Daniel Nephin bd44952c2e streaming: disable streaming when requesting connect events
Until the correct events are created for terminating gateways.
2020-10-26 11:55:49 -04:00
Daniel Nephin c398a6b272 state: disable streaming connect topic 2020-10-26 11:49:47 -04:00
hashicorp-ci 9772c12426 auto-updated agent/uiserver/bindata_assetfs.go from commit 948917c6b 2020-10-26 09:34:58 +00:00
Daniel Nephin 853667e7d8 health: change the name of UseStreamingBackend config
Remove it from the cache section, and update the docs.
2020-10-23 17:47:01 -04:00
Daniel Nephin a62dcc9bfe health: use streaming, even when cache=1 is not set 2020-10-23 17:39:55 -04:00
R.B. Boyer 58387fef0a
server: config entry replication now correctly uses namespaces in comparisons (#9024)
Previously config entries sharing a kind & name but in different
namespaces could occasionally cause "stuck states" in replication
because the namespace fields were ignored during the differential
comparison phase.

Example:

Two config entries written to the primary:

    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo

Under the covers these both get saved to memdb, so they are sorted by
all 3 components (kind,name,namespace) during natural iteration. This
means that before the replication code does it's own incomplete sort,
the underlying data IS sorted by namespace ascending (bar comes before
foo).

After one pass of replication the primary and secondary datacenters have
the same set of config entries present. If
"kind=A,name=web,namespace=bar" were to be deleted, then things get
weird. Before replication the two sides look like:

primary: [
    kind=A,name=web,namespace=foo
]
secondary: [
    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo
]

The differential comparison phase walks these two lists in sorted order
and first compares "kind=A,name=web,namespace=foo" vs
"kind=A,name=web,namespace=bar" and falsely determines they are the SAME
and are thus cause an update of "kind=A,name=web,namespace=foo". Then it
compares "<nothing>" with "kind=A,name=web,namespace=foo" and falsely
determines that the latter should be DELETED.

During reconciliation the deletes are processed before updates, and so
for a brief moment in the secondary "kind=A,name=web,namespace=foo" is
erroneously deleted and then immediately restored.

Unfortunately after this replication phase the final state is identical
to the initial state, so when it loops around again (rate limited) it
repeats the same set of operations indefinitely.
2020-10-23 13:41:54 -05:00
Daniel Nephin 0f1fb24d19 state: convert the remaining functions to ReadTxn
Required also converting some of the transaction functions to WriteTxn
because TxnRO() called the same helper as TxnRW.

This change allows us to return a memdb.Txn for read-only txn instead of
wrapping them with state.txn.
2020-10-23 14:29:22 -04:00
Daniel Nephin 8bd1a2cd16
Merge pull request #8975 from hashicorp/dnephin/stream-close-on-unsub
stream: close the subscription on Unsubscribe
2020-10-23 12:58:12 -04:00
Freddy 9c04cbc40f
Add HasExact to topology endpoint (#9010) 2020-10-23 10:45:41 -06:00
hashicorp-ci 46071cbb03 auto-updated agent/uiserver/bindata_assetfs.go from commit eedee07e7 2020-10-23 16:32:06 +00:00
hashicorp-ci e236a58dba auto-updated agent/uiserver/bindata_assetfs.go from commit 7559f64d0 2020-10-23 14:45:43 +00:00
hashicorp-ci 74b8c024e8 auto-updated agent/uiserver/bindata_assetfs.go from commit 2a8a80638 2020-10-23 13:50:57 +00:00
hashicorp-ci a751688bc9 auto-updated agent/uiserver/bindata_assetfs.go from commit 45554e4e9 2020-10-23 08:32:18 +00:00
R.B. Boyer a2c50d3303
connect: add support for envoy 1.16.0, drop support for 1.12.x, and bump point releases as well (#8944)
Supported versions will be: "1.16.0", "1.15.2", "1.14.5", "1.13.6"
2020-10-22 13:46:19 -05:00
Daniel Nephin 3c52f3854b
Merge pull request #8973 from hashicorp/dnephin/resolver-balance
agent/router: refactor calculation of delay between rebalances.
2020-10-22 13:52:10 -04:00
Daniel Nephin fb57d9b26a stream: close the subscription on Unsubscribe 2020-10-22 13:39:27 -04:00
Daniel Nephin 3a55c30a05
Merge pull request #8924 from ShimmerGlass/fix-sidecar-deregister-after-restart
Fix: service LocallyRegisteredAsSidecar property is not persisted
2020-10-22 13:26:55 -04:00