8e554ee74b
Merge branch 'mkcp/telemetry/add-all-metric-definitions' of ssh://github.com/hashicorp/consul into mkcp/telemetry/add-all-metric-definitions
2020-11-16 15:26:12 -08:00
ad4cebc1d8
fix some tests that were broken from the TelemetryConfig change
2020-11-16 15:22:36 -08:00
fc30f07cc7
linting: sort and group import
2020-11-16 14:17:24 -08:00
2fe021f03c
update runtime_test to handle PrometheusOpts expiry field change
2020-11-16 14:16:12 -08:00
748d56b8ab
Prevent panic if autopilot health is requested prior to leader establishment finishing. ( #9204 )
2020-11-16 17:08:17 -05:00
b81edac7bb
use the MetricsPrefix to set the service name and provide as slice literal to avoid bugs from append modifying its first arg
2020-11-16 14:01:12 -08:00
5e0e4098c9
push prometheus sink definiitons into prometheus.PrometheusOpts
2020-11-16 12:44:47 -08:00
b7367467f6
Merge pull request #9114 from hashicorp/dnephin/filtering-in-stream
...
stream: improve naming of Payload methods
2020-11-16 14:20:07 -05:00
15af5ead0b
trim help strings to save a few bytes
2020-11-16 11:02:11 -08:00
3966ecb02f
merge master
2020-11-16 10:46:53 -08:00
a54d1069b3
auto-updated agent/uiserver/bindata_assetfs.go from commit 959974e96
2020-11-16 15:27:40 +00:00
5da2f1efa8
finish adding static server metrics
2020-11-13 16:26:08 -08:00
16e95f1d7b
Reorganize some CA manager code for correctness/readability
2020-11-13 14:46:01 -08:00
6fba82a4fa
connect: Add CAManager for synchronizing CA operations
2020-11-13 14:33:44 -08:00
af34b26221
connect: Add logic for updating secondary DC intermediate on config set
2020-11-13 14:33:44 -08:00
9eb262252a
server: intentions CRUD requires connect to be enabled ( #9194 )
...
Fixes #9123
2020-11-13 16:19:12 -06:00
06d59c03b9
add the service name in the agent rather than in the definitions themselves
2020-11-13 13:18:04 -08:00
c7233ba871
server: remove config entry CAS in legacy intention API bridge code ( #9151 )
...
Change so line-item intention edits via the API are handled via the state store instead of via CAS operations.
Fixes #9143
2020-11-13 14:42:21 -06:00
c52bc632df
server: skip deleted and deleting namespaces when migrating intentions to config entries ( #9186 )
2020-11-13 13:56:41 -06:00
7af643ac37
ci: update to Go 1.15.4 and alpine:3.12 ( #9036 )
...
* ci: stop building darwin/386 binaries
Go 1.15 drops support for 32-bit binaries on Darwin https://golang.org/doc/go1.15#darwin
* tls: ConnectionState::NegotiatedProtocolIsMutual is deprecated in Go 1.15, this value is always true
* correct error messages that changed slightly
* Completely regenerate some TLS test data
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-11-13 13:02:59 -05:00
fe6b888269
auto-updated agent/uiserver/bindata_assetfs.go from commit 1059a51a3
2020-11-13 16:00:39 +00:00
40cef22c17
auto-updated agent/uiserver/bindata_assetfs.go from commit 78b704be8
2020-11-13 15:44:14 +00:00
c003871c54
server: break up Intention.Apply monolithic method ( #9007 )
...
The Intention.Apply RPC is quite large, so this PR attempts to break it down into smaller functions and dissolves the pre-config-entry approach to the breakdown as it only confused things.
2020-11-13 09:15:39 -06:00
24a2471029
first pass on agent-configured prometheusDefs and adding defs for every consul metric
2020-11-12 18:12:12 -08:00
a397ec85eb
Merge pull request #9162 from hashicorp/dnephin/fix-grpc-metrics
...
grpc: fix metrics
2020-11-12 17:03:01 -05:00
1cedf812e1
auto-updated agent/uiserver/bindata_assetfs.go from commit 6b2970402
2020-11-12 18:49:48 +00:00
61eac21f1a
agent: return the default ACL policy to callers as a header ( #9101 )
...
Header is: X-Consul-Default-ACL-Policy=<allow|deny>
This is of particular utility when fetching matching intentions, as the
fallthrough for a request that doesn't match any intentions is to
enforce using the default acl policy.
2020-11-12 10:38:32 -06:00
4016918434
auto-updated agent/uiserver/bindata_assetfs.go from commit 7243f1f4f
2020-11-12 15:45:53 +00:00
b27457dac8
ci: go-test-race switch to exclude list
...
Most packages should pass the race detector. An exclude list ensures
that new packages are automatically tested with -race.
Also fix a couple small test races to allow more packages to be tested.
Returning readyCh requires a lock because it can be set to nil, and
setting it to nil will race without the lock.
Move the TestServer.Listening calls around so that they properly guard
setting TestServer.l. Otherwise it races.
Remove t.Parallel in a small package. The entire package tests run in a
few seconds, so t.Parallel does very little.
In auto-config, wait for the AutoConfig.run goroutine to stop before
calling readPersistedAutoConfig. Without this change there was a data
race on reading ac.config.
2020-11-11 14:44:57 -05:00
1a137c29d6
grpc: fix grpc metrics
...
defaultMetrics was being set at package import time, which meant that it received an instance of
the original default. But lib/telemetry.InitTelemetry sets a new global when it is called.
This resulted in the metrics being sent nowhere.
This commit changes defaultMetrics to be a function, so it will return the global instance when
called. Since it is called after InitTelemetry it will return the correct metrics instance.
2020-11-11 14:27:25 -05:00
7ef9b04f90
Add a CLI command for retrieving the autopilot configuration. ( #9142 )
2020-11-11 13:19:02 -05:00
848571a73a
auto-updated agent/uiserver/bindata_assetfs.go from commit 6423a2c10
2020-11-11 17:03:36 +00:00
71da0209bf
Add a paramter in state store methods to indicate whether a resource insertion is from a snapshot restoration ( #9156 )
...
The Catalog, Config Entry, KV and Session resources potentially re-validate the input as its coming in. We need to prevent snapshot restoration failures due to missing namespaces or namespaces that are being deleted in enterprise.
2020-11-11 11:21:42 -05:00
37b1ab7f49
auto-updated agent/uiserver/bindata_assetfs.go from commit e1d977138
2020-11-11 14:48:38 +00:00
3760e3d12d
Merge pull request #9149 from joel0/wrap-errors
...
Use error wrapping to preserve error type info
2020-11-10 18:27:08 -05:00
45a9dd59b5
Merge pull request #8976 from joel0/wrap-eof
...
Wrap rpc error object
2020-11-10 17:04:11 -05:00
f600285eb4
Use error wrapping to preserve error type info
2020-11-10 21:50:09 +00:00
77451d944e
auto-updated agent/uiserver/bindata_assetfs.go from commit e18d8e299
2020-11-10 16:37:33 +00:00
8f834c2d21
auto-updated agent/uiserver/bindata_assetfs.go from commit fb6202929
2020-11-10 14:42:02 +00:00
031ab3f44f
auto-updated agent/uiserver/bindata_assetfs.go from commit c8e40ee0d
2020-11-09 17:34:25 +00:00
a3a653342b
Fix a bunch of linter warnings
2020-11-09 09:22:12 -05:00
c048e86bb2
Switch to using the external autopilot module
2020-11-09 09:22:11 -05:00
7a2c6dfd62
auto-updated agent/uiserver/bindata_assetfs.go from commit 5c0ec13fb
2020-11-09 09:31:52 +00:00
908574058e
auto-updated agent/uiserver/bindata_assetfs.go from commit d9672bca8
2020-11-09 09:19:52 +00:00
75019baadd
chore: upgrade to gopsutil/v3 ( #9118 )
...
* deps: update golang.org/x/sys
* deps: update imports to gopsutil/v3
* chore: make update-vendor
2020-11-06 20:48:38 -05:00
fb70c8bac2
stream: document that Payload must be immutable
...
If they are sent to EventPublisher.Publish.
Also document that PayloadEvents is expected to come from a subscription and that it is
not immutable.
2020-11-06 13:00:33 -05:00
8baf158ea8
Revert "Add namespace support for metrics (OSS) ( #9117 )" ( #9124 )
...
This reverts commit 06b3b017d3
2020-11-06 10:24:32 -06:00
cf537ac2f5
auto-updated agent/uiserver/bindata_assetfs.go from commit 3a68686cc
2020-11-06 15:04:29 +00:00
24bc8451d5
auto-updated agent/uiserver/bindata_assetfs.go from commit 848f72f66
2020-11-06 09:31:18 +00:00
06b3b017d3
Add namespace support for metrics (OSS) ( #9117 )
2020-11-05 18:24:29 -07:00
43af0ba7a3
stream: rename FilterByKey
2020-11-05 19:21:16 -05:00
868cfe1eac
stream: Add HasReadPermission to Payload
...
Required now that filter is a method on PayloadEvents instead of Event
2020-11-05 19:17:18 -05:00
36202f7938
stream: move event filtering to PayloadEvents
...
Removes the weirdness around PayloadEvents.FilterByKey
2020-11-05 17:50:17 -05:00
79b5ca1ce6
stream: Remove unused method
2020-11-05 16:49:59 -05:00
8e616a93c1
agent: sanitize ui metrics proxy header values on agent/self endpoint ( #9104 )
2020-11-05 13:25:27 -06:00
a33c50ef0d
Merge pull request #9073 from hashicorp/dnephin/backport-streaming-namespaces
...
streaming: backport namespace changes
2020-11-05 14:19:10 -05:00
c82f6ef2d8
Merge pull request #9061 from hashicorp/dnephin/event-fields
...
stream: support filtering by namespace
2020-11-05 14:18:35 -05:00
977297390c
auto-updated agent/uiserver/bindata_assetfs.go from commit 6ff094976
2020-11-05 19:12:03 +00:00
9d15348565
auto-updated agent/uiserver/bindata_assetfs.go from commit 1ef18c4b6
2020-11-05 16:10:14 +00:00
6ba776b4f3
agent: protect the ui metrics proxy endpoint behind ACLs ( #9099 )
...
This ensures the metrics proxy endpoint is ACL protected behind a
wildcard `service:read` and `node:read` set of rules. For Consul
Enterprise these will need to span all namespaces:
```
service_prefix "" { policy = "read" }
node_prefix "" { policy = "read" }
namespace_prefix "" {
service_prefix "" { policy = "read" }
node_prefix "" { policy = "read" }
}
```
This PR contains just the backend changes. The frontend changes to
actually pass the consul token header to the proxy through the JS plugin
will come in another PR.
2020-11-04 12:50:03 -06:00
a2315bc839
auto-updated agent/uiserver/bindata_assetfs.go from commit 0f6c0a5c1
2020-11-04 09:37:51 +00:00
1a5d4cfe43
auto-updated agent/uiserver/bindata_assetfs.go from commit d5d4155e1
2020-11-03 14:14:58 +00:00
738ff1801f
auto-updated agent/uiserver/bindata_assetfs.go from commit 56c2ff56e
2020-11-02 18:43:31 +00:00
c28f489a9a
auto-updated agent/uiserver/bindata_assetfs.go from commit bf32a1799
2020-11-02 16:11:45 +00:00
907c4ad789
auto-updated agent/uiserver/bindata_assetfs.go from commit 314eeda95
2020-11-02 14:40:27 +00:00
a66c4591d7
agent: introduce path allow list for requests going through the metrics proxy ( #9059 )
...
Added a new option `ui_config.metrics_proxy.path_allowlist`. This defaults to `["/api/v1/query", "/api/v1/query_range"]` when the metrics provider is set to `prometheus`.
Requests that do not use one of the allow-listed paths (via exact match) get a 403 Forbidden response instead.
2020-10-30 16:49:54 -05:00
b532e092dc
structs: add a namespace test for CheckServiceNode.CanRead
2020-10-30 15:07:04 -04:00
9b2fae9bac
cache-type: use namespace in tests
...
to verify that the namespace is passed through correctly to the server.
2020-10-30 15:07:04 -04:00
b95b14e168
state: test EventPayloadCheckServiceNode.FilterByKey
...
Also fix a bug in that function when only one of key or namespace were the empty string.
2020-10-30 14:35:57 -04:00
56d6079da3
stream: Add tests for filterByKey with namespace
...
And fix a bug where a request with a Namespace but no Key would not be properly filtered
2020-10-30 14:35:42 -04:00
2c00045161
stream: Move FilterByKey events to a table
...
In preparation for adding new tests.
2020-10-30 14:35:28 -04:00
43c5803a25
state: use enterprise meta for creating events
2020-10-30 14:34:04 -04:00
0ad2406d7c
stream: include the namespace in the snap cache key
...
Otherwise the wrong snapshot could be returned when the same key is used in different namespaces
2020-10-30 14:34:04 -04:00
c42fe5ae43
subscribe: set the request namespace
2020-10-30 14:34:04 -04:00
b3bf1229ac
auto-updated agent/uiserver/bindata_assetfs.go from commit cf2cfbaf2
2020-10-30 15:27:01 +00:00
fa4b0854fb
state: ensure we unblock intentions queries upon the upgrade to config entries ( #9062 )
...
1. do a state store query to list intentions as the agent would do over in `agent/proxycfg` backing `agent/xds`
2. upgrade the database and do a fresh `service-intentions` config entry write
3. the blocking query inside of the agent cache in (1) doesn't notice (2)
2020-10-29 15:28:31 -05:00
b24b4169e1
restore prior signature of test helper so enterprise compiles
2020-10-29 13:52:15 -05:00
01dbf43fb1
auto-updated agent/uiserver/bindata_assetfs.go from commit 1d6961248
2020-10-29 18:33:41 +00:00
a5dd2001cf
stream: remove Event.Key
...
Makes Payload a type with FilterByKey so that Payloads can implement
filtering by key. With this approach we don't need to expose a Namespace
field on Event, and we don't need to invest micro formats or require a
bunch of code to be aware of exactly how the key field is encoded.
2020-10-28 16:48:04 -04:00
1c094da40d
state: use go-cmp for comparison
...
The output of the previous assertions made it impossible to debug the tests without code changes.
With go-cmp comparing the entire slice we can see the full diffs making it easier to debug failures.
2020-10-28 16:33:00 -04:00
68342a0cb5
proto: remove Event.Key field
...
The field is never used, and the value is available from the payload.
2020-10-28 16:33:00 -04:00
9a1e845be8
proto: remove Event.Namespace field
...
All events are part of a single Topic, so we don't need this field.
2020-10-28 16:33:00 -04:00
3dfb7c224b
stream: Use a no-op event publisher if streaming is disabled
2020-10-28 13:54:19 -04:00
23eee604c9
store: use a ReadDB for snapshots
...
to remove the cyclic dependency between the snapshot handlers and the state.Store
2020-10-28 13:07:42 -04:00
7b9ee25956
Merge pull request #9026 from hashicorp/dnephin/streaming-without-cache-query-param
...
streaming: rename config and remove requirement for cache=1
2020-10-28 12:33:25 -04:00
477d665309
Merge pull request #8618 from hashicorp/dnephin/remove-txn-readtxn
...
state: Use ReadTxn everywhere
2020-10-28 12:32:47 -04:00
9f893307de
Merge pull request #9053 from hashicorp/vault-token-lookupself
...
connect: Use the lookup-self endpoint for Vault token
2020-10-27 14:34:03 -07:00
f0ac093fef
agent/grpc: add connection count metrics
...
Gauge metrics are great for understanding the current state, but can somtimes hide problems
if there are many disconnect/reconnects.
This commit adds counter metrics for connections and streams to make it easier to see the
count of newly created connections and streams.
2020-10-27 16:49:49 -04:00
5319ba02b0
agent/grpc: rename metrics
...
These new names should make it easier to add counter metics with similar prefixes
2020-10-27 16:49:49 -04:00
c82d6aa4ff
Merge pull request #8961 from hashicorp/dnephin/grpc-resolve-node-id
...
agent/grpc: fix some test flakes and handle duplicate server IDs in the pool
2020-10-27 16:47:37 -04:00
74ac34e358
Merge pull request #8998 from hashicorp/dnephin/lib-ttlcache
...
lib/ttlcache: extract a new package from agent/cache
2020-10-27 16:43:10 -04:00
0f81915495
Merge pull request #8987 from hashicorp/dnephin/stream-filter
...
streaming: apply filter to a single item
2020-10-27 16:39:43 -04:00
8bcd5040c7
agent/grpc: Add an integration test for ClientPool with TLS
...
Also deregister the resolver.Builder in tests.
2020-10-27 16:34:18 -04:00
8a785a351c
agent/grpc: pass metrics to constructor
...
Instead of referencing a package var. This does not fix the flaky test, but it seems more correct.
2020-10-27 16:34:17 -04:00
d19657404f
agent/grpc: fix a flaky test by performing more retries
...
Instead of using retry.Run, which appears to have problems in some cases where it does not
emit an error message, use a for loop.
Increase the number of attempts and remove any sleep, since this operation is not that expensive to do
in a tight loop
2020-10-27 16:34:17 -04:00
df405ac978
agent/grpc: remove misleading warnings from test output
...
Handle shutdown properly in tests so that the tests don't warn about using a closed connection.
2020-10-27 16:34:16 -04:00
e101aa8a74
agent/grpc: fix a flake in TestHandler_EmitsStats
2020-10-27 16:34:16 -04:00
19da9c3a9b
agent/grpc: use a separate channel for closing the Accept
...
Closing l.conns can lead to a race and a 'panic: send on closed chan' when a
connection is in the middle of being handled when the server is shutting down.
Found using '-race -count=800'
2020-10-27 16:34:15 -04:00
d8299670cc
agent/grpc/resolver: namespace the server ID with the DC name
...
So that if two datacenters end up with overlapping serverIDs we don't send requests to the wrong server
2020-10-27 16:34:15 -04:00
f700a5707b
connect: Use the lookup-self endpoint for Vault token
2020-10-27 13:03:45 -07:00
Kit Patella
Matt Keeler
Daniel Nephin
hashicorp-ci
Kyle Havlovitz
R.B. Boyer
Mike Morris
Joel May
Matt Keeler
Freddy