Commit Graph

2754 Commits

Author SHA1 Message Date
Hans Hasselberg 57701695c3 add missing descriptions for metrics 2020-11-23 22:06:30 +01:00
Kit Patella fcec25de40 add entries for missing fsm operations and mark duplicated metrics prefixes as deprecated 2020-11-23 12:42:51 -08:00
Kyle Havlovitz 13c31ccfce Clean up the logic in persistNewRootAndConfig 2020-11-20 15:54:44 -08:00
Kit Patella c6b29a8bba
Merge pull request #9245 from hashicorp/telemetry/fix-missing-and-stale-docs
Telemetry/fix missing and stale docs
2020-11-20 12:54:29 -08:00
Pierre Souchay 45151090c1 [Streaming] Predictable order for results of /health/service/:serviceName to mimic memdb
This ensures the result is consitent with/witout streaming

Will partially fix #9239
2020-11-20 16:23:35 +01:00
Kyle Havlovitz 0bfda4481f Add CA server delegate interface for testing 2020-11-19 20:08:06 -08:00
Kit Patella 5c09dc322e add telemetry and definition help entries for missing catalog and acl metrics 2020-11-19 13:29:44 -08:00
R.B. Boyer 7c7a3e5165
command: when generating envoy bootstrap configs use the datacenter returned from the agent services endpoint (#9229)
Fixes #9215
2020-11-19 15:27:31 -06:00
Kit Patella 9e54e897d7 remove stale entries and rename/define acl.resolveToken 2020-11-19 13:06:28 -08:00
hashicorp-ci 8368f65006 auto-updated agent/uiserver/bindata_assetfs.go from commit d913af2bb 2020-11-19 18:45:01 +00:00
Freddy fd5928fa4e
Require operator:write to get Connect CA config (#9240)
A vulnerability was identified in Consul and Consul Enterprise (“Consul”) such that operators with `operator:read` ACL permissions are able to read the Consul Connect CA configuration when explicitly configured with the `/v1/connect/ca/configuration` endpoint, including the private key. This allows the user to effectively privilege escalate by enabling the ability to mint certificates for any Consul Connect services. This would potentially allow them to masquerade (receive/send traffic) as any service in the mesh.

--

This PR increases the permissions required to read the Connect CA's private key when it was configured via the `/connect/ca/configuration` endpoint. They are now `operator:write`.
2020-11-19 10:14:48 -07:00
hashicorp-ci effe235562 auto-updated agent/uiserver/bindata_assetfs.go from commit 687ce1f9c 2020-11-19 16:13:04 +00:00
Daniel Nephin 671b8cf494
Merge pull request #9224 from hashicorp/dnephin/fix-multiple-http-listeners
agent: fix bug with multiple listeners
2020-11-18 16:52:29 -05:00
Daniel Nephin 79963be559 Use freeport
To prevent other tests which already use freeport from flaking when port 0 steals their reserved port.
2020-11-18 16:07:34 -05:00
hashicorp-ci b8659f77c4 auto-updated agent/uiserver/bindata_assetfs.go from commit 591a96d5b 2020-11-18 19:07:25 +00:00
hashicorp-ci cd003a14a5 auto-updated agent/uiserver/bindata_assetfs.go from commit 1edef424a 2020-11-18 19:00:19 +00:00
Daniel Nephin 738bf9efdc agent: fix bug with multiple listeners
Previously the listener was being passed to a closure in a loop without
capturing the loop variable. The result is only the last listener is
used, so the http/https servers only listen on one address.

This problem is fixed by capturing the variable by passing it into a
function.
2020-11-18 13:03:29 -05:00
hashicorp-ci b2605d90d2 auto-updated agent/uiserver/bindata_assetfs.go from commit 664f1d9aa 2020-11-18 11:17:06 +00:00
Kyle Havlovitz 9be7c6401c connect: update some function comments in CA manager 2020-11-17 16:00:19 -08:00
Daniel Nephin 839429eb40
Merge pull request #9160 from hashicorp/dnephin/go-test-race-in-to-out-list
ci: change go-test-race package list to exclude list
2020-11-17 13:13:38 -05:00
Matt Keeler 66fd23d67f
Refactor to call non-voting servers read replicas (#9191)
Co-authored-by: Kit Patella <kit@jepsen.io>
2020-11-17 10:53:57 -05:00
Kit Patella d15b6fddd3
Merge pull request #9198 from hashicorp/mkcp/telemetry/add-all-metric-definitions
Add metric definitions for all metrics known at Consul start
2020-11-16 15:54:50 -08:00
hashicorp-ci 56dbabf67b auto-updated agent/uiserver/bindata_assetfs.go from commit fe728855e 2020-11-16 23:41:31 +00:00
Freddy fe728855ed
Add DC and NS support for Envoy metrics (#9207)
This PR updates the tags that we generate for Envoy stats.

Several of these come with breaking changes, since we can't keep two stats prefixes for a filter.
2020-11-16 16:37:19 -07:00
Kit Patella 8e554ee74b Merge branch 'mkcp/telemetry/add-all-metric-definitions' of ssh://github.com/hashicorp/consul into mkcp/telemetry/add-all-metric-definitions 2020-11-16 15:26:12 -08:00
Kit Patella ad4cebc1d8 fix some tests that were broken from the TelemetryConfig change 2020-11-16 15:22:36 -08:00
Kit Patella fc30f07cc7
linting: sort and group import 2020-11-16 14:17:24 -08:00
Kit Patella 2fe021f03c update runtime_test to handle PrometheusOpts expiry field change 2020-11-16 14:16:12 -08:00
Matt Keeler 748d56b8ab
Prevent panic if autopilot health is requested prior to leader establishment finishing. (#9204) 2020-11-16 17:08:17 -05:00
Kit Patella b81edac7bb use the MetricsPrefix to set the service name and provide as slice literal to avoid bugs from append modifying its first arg 2020-11-16 14:01:12 -08:00
Kit Patella 5e0e4098c9 push prometheus sink definiitons into prometheus.PrometheusOpts 2020-11-16 12:44:47 -08:00
Daniel Nephin b7367467f6
Merge pull request #9114 from hashicorp/dnephin/filtering-in-stream
stream: improve naming of Payload methods
2020-11-16 14:20:07 -05:00
Kit Patella 15af5ead0b trim help strings to save a few bytes 2020-11-16 11:02:11 -08:00
Kit Patella 3966ecb02f merge master 2020-11-16 10:46:53 -08:00
hashicorp-ci a54d1069b3 auto-updated agent/uiserver/bindata_assetfs.go from commit 959974e96 2020-11-16 15:27:40 +00:00
Kit Patella 5da2f1efa8 finish adding static server metrics 2020-11-13 16:26:08 -08:00
Kyle Havlovitz 16e95f1d7b Reorganize some CA manager code for correctness/readability 2020-11-13 14:46:01 -08:00
Kyle Havlovitz 6fba82a4fa connect: Add CAManager for synchronizing CA operations 2020-11-13 14:33:44 -08:00
Kyle Havlovitz af34b26221 connect: Add logic for updating secondary DC intermediate on config set 2020-11-13 14:33:44 -08:00
R.B. Boyer 9eb262252a
server: intentions CRUD requires connect to be enabled (#9194)
Fixes #9123
2020-11-13 16:19:12 -06:00
Kit Patella 06d59c03b9 add the service name in the agent rather than in the definitions themselves 2020-11-13 13:18:04 -08:00
R.B. Boyer c7233ba871
server: remove config entry CAS in legacy intention API bridge code (#9151)
Change so line-item intention edits via the API are handled via the state store instead of via CAS operations.

Fixes #9143
2020-11-13 14:42:21 -06:00
R.B. Boyer c52bc632df
server: skip deleted and deleting namespaces when migrating intentions to config entries (#9186) 2020-11-13 13:56:41 -06:00
Mike Morris 7af643ac37
ci: update to Go 1.15.4 and alpine:3.12 (#9036)
* ci: stop building darwin/386 binaries

Go 1.15 drops support for 32-bit binaries on Darwin https://golang.org/doc/go1.15#darwin

* tls: ConnectionState::NegotiatedProtocolIsMutual is deprecated in Go 1.15, this value is always true

* correct error messages that changed slightly

* Completely regenerate some TLS test data

Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-11-13 13:02:59 -05:00
hashicorp-ci fe6b888269 auto-updated agent/uiserver/bindata_assetfs.go from commit 1059a51a3 2020-11-13 16:00:39 +00:00
hashicorp-ci 40cef22c17 auto-updated agent/uiserver/bindata_assetfs.go from commit 78b704be8 2020-11-13 15:44:14 +00:00
R.B. Boyer c003871c54
server: break up Intention.Apply monolithic method (#9007)
The Intention.Apply RPC is quite large, so this PR attempts to break it down into smaller functions and dissolves the pre-config-entry approach to the breakdown as it only confused things.
2020-11-13 09:15:39 -06:00
Kit Patella 24a2471029 first pass on agent-configured prometheusDefs and adding defs for every consul metric 2020-11-12 18:12:12 -08:00
Daniel Nephin a397ec85eb
Merge pull request #9162 from hashicorp/dnephin/fix-grpc-metrics
grpc: fix metrics
2020-11-12 17:03:01 -05:00
hashicorp-ci 1cedf812e1 auto-updated agent/uiserver/bindata_assetfs.go from commit 6b2970402 2020-11-12 18:49:48 +00:00
R.B. Boyer 61eac21f1a
agent: return the default ACL policy to callers as a header (#9101)
Header is: X-Consul-Default-ACL-Policy=<allow|deny>

This is of particular utility when fetching matching intentions, as the
fallthrough for a request that doesn't match any intentions is to
enforce using the default acl policy.
2020-11-12 10:38:32 -06:00
hashicorp-ci 4016918434 auto-updated agent/uiserver/bindata_assetfs.go from commit 7243f1f4f 2020-11-12 15:45:53 +00:00
Daniel Nephin b27457dac8 ci: go-test-race switch to exclude list
Most packages should pass the race detector. An exclude list ensures
that new packages are automatically tested with -race.

Also fix a couple small test races to allow more packages to be tested.

Returning readyCh requires a lock because it can be set to nil, and
setting it to nil will race without the lock.

Move the TestServer.Listening calls around so that they properly guard
setting TestServer.l. Otherwise it races.

Remove t.Parallel in a small package. The entire package tests run in a
few seconds, so t.Parallel does very little.

In auto-config, wait for the AutoConfig.run goroutine to stop before
calling readPersistedAutoConfig. Without this change there was a data
race on reading ac.config.
2020-11-11 14:44:57 -05:00
Daniel Nephin 1a137c29d6 grpc: fix grpc metrics
defaultMetrics was being set at package import time, which meant that it received an instance of
the original default. But lib/telemetry.InitTelemetry sets a new global when it is called.

This resulted in the metrics being sent nowhere.

This commit changes defaultMetrics to be a function, so it will return the global instance when
called. Since it is called after InitTelemetry it will return the correct metrics instance.
2020-11-11 14:27:25 -05:00
Matt Keeler 7ef9b04f90
Add a CLI command for retrieving the autopilot configuration. (#9142) 2020-11-11 13:19:02 -05:00
hashicorp-ci 848571a73a auto-updated agent/uiserver/bindata_assetfs.go from commit 6423a2c10 2020-11-11 17:03:36 +00:00
Matt Keeler 71da0209bf
Add a paramter in state store methods to indicate whether a resource insertion is from a snapshot restoration (#9156)
The Catalog, Config Entry, KV and Session resources potentially re-validate the input as its coming in. We need to prevent snapshot restoration failures due to missing namespaces or namespaces that are being deleted in enterprise.
2020-11-11 11:21:42 -05:00
hashicorp-ci 37b1ab7f49 auto-updated agent/uiserver/bindata_assetfs.go from commit e1d977138 2020-11-11 14:48:38 +00:00
Daniel Nephin 3760e3d12d
Merge pull request #9149 from joel0/wrap-errors
Use error wrapping to preserve error type info
2020-11-10 18:27:08 -05:00
Daniel Nephin 45a9dd59b5
Merge pull request #8976 from joel0/wrap-eof
Wrap rpc error object
2020-11-10 17:04:11 -05:00
Joel May f600285eb4 Use error wrapping to preserve error type info 2020-11-10 21:50:09 +00:00
hashicorp-ci 77451d944e auto-updated agent/uiserver/bindata_assetfs.go from commit e18d8e299 2020-11-10 16:37:33 +00:00
hashicorp-ci 8f834c2d21 auto-updated agent/uiserver/bindata_assetfs.go from commit fb6202929 2020-11-10 14:42:02 +00:00
hashicorp-ci 031ab3f44f auto-updated agent/uiserver/bindata_assetfs.go from commit c8e40ee0d 2020-11-09 17:34:25 +00:00
Matt Keeler a3a653342b
Fix a bunch of linter warnings 2020-11-09 09:22:12 -05:00
Matt Keeler c048e86bb2
Switch to using the external autopilot module 2020-11-09 09:22:11 -05:00
hashicorp-ci 7a2c6dfd62 auto-updated agent/uiserver/bindata_assetfs.go from commit 5c0ec13fb 2020-11-09 09:31:52 +00:00
hashicorp-ci 908574058e auto-updated agent/uiserver/bindata_assetfs.go from commit d9672bca8 2020-11-09 09:19:52 +00:00
Mike Morris 75019baadd
chore: upgrade to gopsutil/v3 (#9118)
* deps: update golang.org/x/sys

* deps: update imports to gopsutil/v3

* chore: make update-vendor
2020-11-06 20:48:38 -05:00
Daniel Nephin fb70c8bac2 stream: document that Payload must be immutable
If they are sent to EventPublisher.Publish.

Also document that PayloadEvents is expected to come from a subscription and that it is
not immutable.
2020-11-06 13:00:33 -05:00
R.B. Boyer 8baf158ea8
Revert "Add namespace support for metrics (OSS) (#9117)" (#9124)
This reverts commit 06b3b017d3.
2020-11-06 10:24:32 -06:00
hashicorp-ci cf537ac2f5 auto-updated agent/uiserver/bindata_assetfs.go from commit 3a68686cc 2020-11-06 15:04:29 +00:00
hashicorp-ci 24bc8451d5 auto-updated agent/uiserver/bindata_assetfs.go from commit 848f72f66 2020-11-06 09:31:18 +00:00
Freddy 06b3b017d3
Add namespace support for metrics (OSS) (#9117) 2020-11-05 18:24:29 -07:00
Daniel Nephin 43af0ba7a3 stream: rename FilterByKey 2020-11-05 19:21:16 -05:00
Daniel Nephin 868cfe1eac stream: Add HasReadPermission to Payload
Required now that filter is a method on PayloadEvents instead of Event
2020-11-05 19:17:18 -05:00
Daniel Nephin 36202f7938 stream: move event filtering to PayloadEvents
Removes the weirdness around PayloadEvents.FilterByKey
2020-11-05 17:50:17 -05:00
Daniel Nephin 79b5ca1ce6 stream: Remove unused method 2020-11-05 16:49:59 -05:00
R.B. Boyer 8e616a93c1
agent: sanitize ui metrics proxy header values on agent/self endpoint (#9104) 2020-11-05 13:25:27 -06:00
Daniel Nephin a33c50ef0d
Merge pull request #9073 from hashicorp/dnephin/backport-streaming-namespaces
streaming: backport namespace changes
2020-11-05 14:19:10 -05:00
Daniel Nephin c82f6ef2d8
Merge pull request #9061 from hashicorp/dnephin/event-fields
stream: support filtering by namespace
2020-11-05 14:18:35 -05:00
hashicorp-ci 977297390c auto-updated agent/uiserver/bindata_assetfs.go from commit 6ff094976 2020-11-05 19:12:03 +00:00
hashicorp-ci 9d15348565 auto-updated agent/uiserver/bindata_assetfs.go from commit 1ef18c4b6 2020-11-05 16:10:14 +00:00
R.B. Boyer 6ba776b4f3
agent: protect the ui metrics proxy endpoint behind ACLs (#9099)
This ensures the metrics proxy endpoint is ACL protected behind a
wildcard `service:read` and `node:read` set of rules. For Consul
Enterprise these will need to span all namespaces:

```
service_prefix "" { policy = "read" }
node_prefix ""    { policy = "read" }

namespace_prefix "" {
  service_prefix "" { policy = "read" }
  node_prefix ""    { policy = "read" }
}
```

This PR contains just the backend changes. The frontend changes to
actually pass the consul token header to the proxy through the JS plugin
will come in another PR.
2020-11-04 12:50:03 -06:00
hashicorp-ci a2315bc839 auto-updated agent/uiserver/bindata_assetfs.go from commit 0f6c0a5c1 2020-11-04 09:37:51 +00:00
hashicorp-ci 1a5d4cfe43 auto-updated agent/uiserver/bindata_assetfs.go from commit d5d4155e1 2020-11-03 14:14:58 +00:00
hashicorp-ci 738ff1801f auto-updated agent/uiserver/bindata_assetfs.go from commit 56c2ff56e 2020-11-02 18:43:31 +00:00
hashicorp-ci c28f489a9a auto-updated agent/uiserver/bindata_assetfs.go from commit bf32a1799 2020-11-02 16:11:45 +00:00
hashicorp-ci 907c4ad789 auto-updated agent/uiserver/bindata_assetfs.go from commit 314eeda95 2020-11-02 14:40:27 +00:00
R.B. Boyer a66c4591d7
agent: introduce path allow list for requests going through the metrics proxy (#9059)
Added a new option `ui_config.metrics_proxy.path_allowlist`. This defaults to `["/api/v1/query", "/api/v1/query_range"]` when the metrics provider is set to `prometheus`.

Requests that do not use one of the allow-listed paths (via exact match) get a 403 Forbidden response instead.
2020-10-30 16:49:54 -05:00
Daniel Nephin b532e092dc structs: add a namespace test for CheckServiceNode.CanRead 2020-10-30 15:07:04 -04:00
Daniel Nephin 9b2fae9bac cache-type: use namespace in tests
to verify that the namespace is passed through correctly to the server.
2020-10-30 15:07:04 -04:00
Daniel Nephin b95b14e168 state: test EventPayloadCheckServiceNode.FilterByKey
Also fix a bug in that function when only one of key or namespace were the empty string.
2020-10-30 14:35:57 -04:00
Daniel Nephin 56d6079da3 stream: Add tests for filterByKey with namespace
And fix a bug where a request with a Namespace but no Key would not be properly filtered
2020-10-30 14:35:42 -04:00
Daniel Nephin 2c00045161 stream: Move FilterByKey events to a table
In preparation for adding new tests.
2020-10-30 14:35:28 -04:00
Daniel Nephin 43c5803a25 state: use enterprise meta for creating events 2020-10-30 14:34:04 -04:00
Daniel Nephin 0ad2406d7c stream: include the namespace in the snap cache key
Otherwise the wrong snapshot could be returned when the same key is used in different namespaces
2020-10-30 14:34:04 -04:00
Daniel Nephin c42fe5ae43 subscribe: set the request namespace 2020-10-30 14:34:04 -04:00
hashicorp-ci b3bf1229ac auto-updated agent/uiserver/bindata_assetfs.go from commit cf2cfbaf2 2020-10-30 15:27:01 +00:00
R.B. Boyer fa4b0854fb
state: ensure we unblock intentions queries upon the upgrade to config entries (#9062)
1. do a state store query to list intentions as the agent would do over in `agent/proxycfg` backing `agent/xds`
2. upgrade the database and do a fresh `service-intentions` config entry write
3. the blocking query inside of the agent cache in (1) doesn't notice (2)
2020-10-29 15:28:31 -05:00