Commit Graph

3188 Commits

Author SHA1 Message Date
Daniel Nephin a04cefaa28 Remove an unnecessary else 2021-01-07 18:13:49 -05:00
Daniel Nephin 4b8b2a4291 xds: remove Server.Initialize
Requiring a call to initialize to set a single field is not really substantially different
from having to set that field to a value.
2021-01-07 18:13:48 -05:00
Daniel Nephin 2e2ee41390 xds: Fix data race
TestEnvoy.Close used e.stream.recvCh == nil to indicate the channel had already
been closed, so that TestEnvoy.Close can be called multiple times. The recvCh
was not protected by a lock, so setting it to nil caused a data race with any
goroutine trying to read from the channel.

Instead set the stream to nil. The stream is guarded by a lock, so it does not race.

This change allows us to test the agent/xds package using -race.
2021-01-07 18:13:48 -05:00
Daniel Nephin 375aed5ed6 xds: Pass in logger
small cleanup in tests
2021-01-07 18:13:48 -05:00
hashicorp-ci a040495cd3 auto-updated agent/uiserver/bindata_assetfs.go from commit e893ba7ea 2021-01-07 19:09:58 +00:00
Daniel Nephin d64425d2e4
Merge pull request from hashicorp/dnephin/resolve-tokens-take-2
acl: Remove some unused things and document delegate method
2021-01-06 18:51:51 -05:00
Pierre Souchay 994fe80358 Added testing of GRPC with TLS combinations
This ensures that https://github.com/hashicorp/consul/issues/9474 will
not reproduce.
2021-01-06 22:20:23 +01:00
Pierre Souchay e2f2d4b0d7 [Streaming][bugfix] handle TLS signalisation when TLS is disabled on client side
Tnis is an alternative to https://github.com/hashicorp/consul/pull/9494
2021-01-06 17:24:58 +01:00
R.B. Boyer 19baf4bc25
acl: use the presence of a management policy in the state store as a sign that we already migrated to v2 acls ()
This way we only have to wait for the serf barrier to pass once before
we can upgrade to v2 acls. Without this patch every restart needs to
re-compute the change, and potentially if a stray older node joins after
a migration it might regress back to v1 mode which would be problematic.
2021-01-05 17:04:27 -06:00
hashicorp-ci 2e67ee728c auto-updated agent/uiserver/bindata_assetfs.go from commit 1304dc882 2021-01-05 17:47:53 +00:00
Daniel Nephin b0574c9839
Merge pull request from naemono/6074-allow-config-MaxHeaderBytes
Adds option to configure HTTP Server's MaxHeaderBytes
2021-01-05 12:28:27 -05:00
hashicorp-ci a30959d5ae auto-updated agent/uiserver/bindata_assetfs.go from commit a42e844cc 2021-01-05 17:09:19 +00:00
hashicorp-ci 3925713767 auto-updated agent/uiserver/bindata_assetfs.go from commit 17438020f 2021-01-05 10:11:12 +00:00
Michael Montgomery ba3fe0c875 Remove unneeded test 2021-01-04 19:47:13 -06:00
Matt Keeler 85e5da53d5
Special case the error returned when we have a Raft leader but are not tracking it in the ServerLookup ()
This can happen when one other node in the cluster such as a client is unable to communicate with the leader server and sees it as failed. When that happens its failing status eventually gets propagated to the other servers in the cluster and eventually this can result in RPCs returning “No cluster leader” error.

That error is misleading and unhelpful for determing the root cause of the issue as its not raft stability but rather and client -> server networking issue. Therefore this commit will add a new error that will be returned in that case to differentiate between the two cases.
2021-01-04 14:05:23 -05:00
hashicorp-ci a5bb8fc52e auto-updated agent/uiserver/bindata_assetfs.go from commit 8c9d5ecc2 2021-01-04 18:36:22 +00:00
R.B. Boyer d5d62d9e08
server: deletions of intentions by name using the intention API is now idempotent ()
Restoring a behavior inadvertently changed while fixing 
2021-01-04 11:27:00 -06:00
hashicorp-ci e66f02541e auto-updated agent/uiserver/bindata_assetfs.go from commit 8c0473a62 2021-01-04 16:52:35 +00:00
Michael Montgomery e4f603dfae Merge branch 'master' into 6074-allow-config-MaxHeaderBytes 2020-12-30 14:14:05 -06:00
Michael Montgomery 817903b925 Fixed failing tests
Removed use of `NewTestAgent`, per review comment
Removed CLI flag, per review comment
Updated website documentation
Added changelog entry
2020-12-30 14:09:50 -06:00
Daniel Nephin 71b82a7e5b Maybe fix another data race in a test 2020-12-22 18:53:54 -05:00
Daniel Nephin bae9125fc1 Fix one race caused by t.Parallel 2020-12-22 18:27:18 -05:00
hashicorp-ci cacbff9a50 auto-updated agent/uiserver/bindata_assetfs.go from commit 99f102705 2020-12-18 10:43:59 +00:00
hashicorp-ci 9ca4e431f6 auto-updated agent/uiserver/bindata_assetfs.go from commit 25d6a1277 2020-12-18 09:07:19 +00:00
hashicorp-ci 9fc99f4ae0 auto-updated agent/uiserver/bindata_assetfs.go from commit c7d917777 2020-12-17 19:01:42 +00:00
hashicorp-ci a05be282ad auto-updated agent/uiserver/bindata_assetfs.go from commit 2e3a66efb 2020-12-17 16:39:34 +00:00
hashicorp-ci 1ecece3ee8 auto-updated agent/uiserver/bindata_assetfs.go from commit 635cf4dc9 2020-12-17 16:08:30 +00:00
hashicorp-ci 8e35646ba1 auto-updated agent/uiserver/bindata_assetfs.go from commit 66cc91c69 2020-12-17 15:30:54 +00:00
hashicorp-ci 744d467540 auto-updated agent/uiserver/bindata_assetfs.go from commit 0ca54c608 2020-12-16 16:46:08 +00:00
hashicorp-ci c086f5eecb auto-updated agent/uiserver/bindata_assetfs.go from commit 4404b4f44 2020-12-16 09:23:22 +00:00
hashicorp-ci eb8c46c5bb auto-updated agent/uiserver/bindata_assetfs.go from commit a919b60f5 2020-12-15 19:36:10 +00:00
hashicorp-ci 3fa218a4d2 auto-updated agent/uiserver/bindata_assetfs.go from commit e921b3cf9 2020-12-15 18:33:36 +00:00
hashicorp-ci 2cf057db55 auto-updated agent/uiserver/bindata_assetfs.go from commit 5e150d7f0 2020-12-15 18:19:30 +00:00
John Cowen a9913b5b22
api: Ensure the internal/ui/service endpoint responds with an array ()
In some circumstances this endpoint will have no results in it (dues to
ACLs, Namespaces or filtering).

This ensures that the response is at least an empty array (`[]`) rather
than `null`
2020-12-15 16:52:00 +00:00
hashicorp-ci 7830eb85f6 auto-updated agent/uiserver/bindata_assetfs.go from commit 9f0787197 2020-12-15 16:37:21 +00:00
hashicorp-ci 57899ba5ef auto-updated agent/uiserver/bindata_assetfs.go from commit f111d6b3e 2020-12-15 15:40:26 +00:00
hashicorp-ci 35678670cd auto-updated agent/uiserver/bindata_assetfs.go from commit 14d043e5f 2020-12-14 15:33:47 +00:00
hashicorp-ci 0c3db76c92 auto-updated agent/uiserver/bindata_assetfs.go from commit 4e419b9b3 2020-12-14 14:31:06 +00:00
Daniel Nephin c15d5b0ed1 http: Check HTTPUseCache in a single place
HTTPUseCache is only used is a gate for allowing QueryOptions.UseCache to be enabled. By
moving it to the place where the query options are set, this behaviour is more obvious.

Also remove parseInternal which was an alias for parse.
2020-12-11 14:03:47 -05:00
Daniel Nephin cb3dbc92f9
Merge pull request from hashicorp/dnephin/skip-slow-tests-with-short
testing: skip slow tests with -short
2020-12-11 13:33:44 -05:00
hashicorp-ci 2608e92763 auto-updated agent/uiserver/bindata_assetfs.go from commit 514270a41 2020-12-11 11:48:26 +00:00
hashicorp-ci db175eaf38 auto-updated agent/uiserver/bindata_assetfs.go from commit 0f7c909f1 2020-12-11 09:44:56 +00:00
hashicorp-ci 7deddd82c8 auto-updated agent/uiserver/bindata_assetfs.go from commit 21e5a8f0f 2020-12-11 09:38:09 +00:00
R.B. Boyer d921690bfe
acl: global tokens created by auth methods now correctly replicate to secondary datacenters ()
Previously the tokens would fail to insert into the secondary's state
store because the AuthMethod field of the ACLToken did not point to a
known auth method from the primary.
2020-12-09 15:22:29 -06:00
hashicorp-ci 673158416d auto-updated agent/uiserver/bindata_assetfs.go from commit 9d8131907 2020-12-09 19:17:30 +00:00
hashicorp-ci 4dbc1b7b59 auto-updated agent/uiserver/bindata_assetfs.go from commit a78566e2d 2020-12-09 18:54:02 +00:00
hashicorp-ci bb8e53bbf7 auto-updated agent/uiserver/bindata_assetfs.go from commit d6cb2b0d7 2020-12-09 18:45:39 +00:00
Kenia 27f6899ec8
Create consul version metric with version label ()
* create consul version metric with version label

* agent/agent.go: add pre-release Version as well as label

Co-Authored-By: Radha13 <kumari.radha3@gmail.com>

* verion and pre-release version labels.

* hyphen/- breaks prometheus

* Add Prometheus gauge defintion for version metric

* Add new metric to telemetry docs

Co-authored-by: Radha Kumari <kumari.radha3@gmail.com>
Co-authored-by: Aestek <thib.gilles@gmail.com>
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2020-12-09 09:16:53 -05:00
hashicorp-ci bad0ada7cb auto-updated agent/uiserver/bindata_assetfs.go from commit 27c74f714 2020-12-09 13:12:32 +00:00
hashicorp-ci 4387740f01 auto-updated agent/uiserver/bindata_assetfs.go from commit 613be01f4 2020-12-09 09:29:17 +00:00
hashicorp-ci 67f7511ea8 auto-updated agent/uiserver/bindata_assetfs.go from commit db5283ee2 2020-12-08 15:53:31 +00:00
hashicorp-ci 9a212f13b7 auto-updated agent/uiserver/bindata_assetfs.go from commit 3be03029f 2020-12-08 09:32:32 +00:00
Daniel Nephin b9e60c0775 testing: skip slow tests with -short
Add a skip condition to all tests slower than 100ms.

This change was made using `gotestsum tool slowest` with data from the
last 3 CI runs of master.
See https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests

With this change:

```
$ time go test -count=1 -short ./agent
ok      github.com/hashicorp/consul/agent       0.743s

real    0m4.791s

$ time go test -count=1 -short ./agent/consul
ok      github.com/hashicorp/consul/agent/consul        4.229s

real    0m8.769s
```
2020-12-07 13:42:55 -05:00
hashicorp-ci 0d35351e2d auto-updated agent/uiserver/bindata_assetfs.go from commit 4dfa7622d 2020-12-07 09:24:00 +00:00
hashicorp-ci 76d5d98b5e auto-updated agent/uiserver/bindata_assetfs.go from commit adbd6c0c8 2020-12-03 09:19:50 +00:00
Kyle Havlovitz 88d669c0e0 connect: Fix a case where the active root would get unset even when there wasn't a new one 2020-12-02 11:42:23 -08:00
hashicorp-ci de2b3a11dd auto-updated agent/uiserver/bindata_assetfs.go from commit e23b5b003 2020-12-02 15:53:16 +00:00
hashicorp-ci b08fb88a94 auto-updated agent/uiserver/bindata_assetfs.go from commit 9ac7bc180 2020-12-02 15:46:59 +00:00
hashicorp-ci e49bf629f8 auto-updated agent/uiserver/bindata_assetfs.go from commit a5b9ada9a 2020-12-02 09:49:40 +00:00
hashicorp-ci 2eedfd57e7 auto-updated agent/uiserver/bindata_assetfs.go from commit cf38309f6 2020-12-01 15:49:06 +00:00
Kyle Havlovitz c4eff420be
Merge pull request from hashicorp/update-secondary-ca
connect: Fix an issue with updating CA config in a secondary datacenter
2020-11-30 14:49:28 -08:00
Kyle Havlovitz 781cae5809 Use a buffered channel for CA intermediate renew func 2020-11-30 14:37:24 -08:00
hashicorp-ci 4279a2d7d4 auto-updated agent/uiserver/bindata_assetfs.go from commit afe0f2614 2020-11-30 18:47:37 +00:00
hashicorp-ci 70666c22bb auto-updated agent/uiserver/bindata_assetfs.go from commit b5abbf122 2020-11-30 17:33:21 +00:00
hashicorp-ci 9b16766ffe auto-updated agent/uiserver/bindata_assetfs.go from commit d1ebe8c14 2020-11-30 17:27:35 +00:00
hashicorp-ci 2a78aefc5a auto-updated agent/uiserver/bindata_assetfs.go from commit f46ef3e3f 2020-11-30 17:07:25 +00:00
hashicorp-ci cb4c3b83b9 auto-updated agent/uiserver/bindata_assetfs.go from commit a59a2f860 2020-11-30 16:57:34 +00:00
hashicorp-ci 5f5ac36375 auto-updated agent/uiserver/bindata_assetfs.go from commit 9cf30e74e 2020-11-30 15:09:43 +00:00
Daniel Nephin 9f0f2bd589
Merge pull request from hashicorp/dnephin/agent-service-register
local: mark service as InSync when added to local agent state
2020-11-27 15:49:55 -05:00
Daniel Nephin 33b81067f8 local: mark service and checks as InSync when added
If the existing service and checks are the same as the new registration.
2020-11-27 15:31:12 -05:00
Hans Hasselberg 44674bcdf8
fix serf_wan documentation ()
WAN config is different than LAN config, source of truth is
f72d2042a8/config.go (L315-L326)
and now the docs are correct.
2020-11-27 20:49:43 +01:00
hashicorp-ci a1644351bc auto-updated agent/uiserver/bindata_assetfs.go from commit 408174f3b 2020-11-27 15:45:17 +00:00
Daniel Nephin 08b8a9276d
Merge pull request from pierresouchay/streaming_predictible_order_for_health
[Streaming] Predictable order for results of /health/service/:serviceName to mimic memdb
2020-11-25 15:53:18 -05:00
Pierre Souchay 76d95fd388 Applied suggestions from @dnephin
* Renamed `cachedHealResultSorter` into `sortCheckServiceNodes`
* Use `<` instead of `strings.Compare`
* Single line comparison in unit test
2020-11-25 21:40:51 +01:00
R.B. Boyer d2d1b05a4e
server: fix panic when deleting a non existent intention ()
* server: fix panic when deleting a non existent intention

* add changelog

* Always return an error when deleting non-existent ixn

Co-authored-by: freddygv <gh@freddygv.xyz>
2020-11-24 13:44:20 -05:00
hashicorp-ci 9ccf12289a auto-updated agent/uiserver/bindata_assetfs.go from commit 6f8b5acbe 2020-11-24 17:51:46 +00:00
hashicorp-ci 3f9d15959c auto-updated agent/uiserver/bindata_assetfs.go from commit 9c3c7bcf3 2020-11-24 14:38:24 +00:00
Hans Hasselberg 57701695c3 add missing descriptions for metrics 2020-11-23 22:06:30 +01:00
Kit Patella fcec25de40 add entries for missing fsm operations and mark duplicated metrics prefixes as deprecated 2020-11-23 12:42:51 -08:00
Daniel Nephin a1e4b69527 config: remove unused const 2020-11-20 19:17:12 -05:00
Kyle Havlovitz 13c31ccfce Clean up the logic in persistNewRootAndConfig 2020-11-20 15:54:44 -08:00
Daniel Nephin 78f767e78c config: move testing shims to BuilderOpts
And remove the devMode field from builder.

This change helps make the Builder state more explicit by moving inputs to the BuilderOps struct,
leaving only fields that can change during Builder.Build on the Builder struct.
2020-11-20 18:31:10 -05:00
Daniel Nephin 0a44906fe3 config: Use LiteralSource for some defaults
Using the LiteralSource makes it much easier to find default values, because an IDE reports
the location of a default. With an HCL string they are harder to discover.

Also removes unnecessary mapstructure.Decodes of constant values.
2020-11-20 18:14:17 -05:00
Kit Patella c6b29a8bba
Merge pull request from hashicorp/telemetry/fix-missing-and-stale-docs
Telemetry/fix missing and stale docs
2020-11-20 12:54:29 -08:00
Pierre Souchay 45151090c1 [Streaming] Predictable order for results of /health/service/:serviceName to mimic memdb
This ensures the result is consitent with/witout streaming

Will partially fix 
2020-11-20 16:23:35 +01:00
Michael Montgomery 585c84e9ff Merge branch 'master' into 6074-allow-config-MaxHeaderBytes 2020-11-20 07:43:53 -06:00
Kyle Havlovitz 0bfda4481f Add CA server delegate interface for testing 2020-11-19 20:08:06 -08:00
Kit Patella 5c09dc322e add telemetry and definition help entries for missing catalog and acl metrics 2020-11-19 13:29:44 -08:00
R.B. Boyer 7c7a3e5165
command: when generating envoy bootstrap configs use the datacenter returned from the agent services endpoint ()
Fixes 
2020-11-19 15:27:31 -06:00
Kit Patella 9e54e897d7 remove stale entries and rename/define acl.resolveToken 2020-11-19 13:06:28 -08:00
hashicorp-ci 8368f65006 auto-updated agent/uiserver/bindata_assetfs.go from commit d913af2bb 2020-11-19 18:45:01 +00:00
Freddy fd5928fa4e
Require operator:write to get Connect CA config ()
A vulnerability was identified in Consul and Consul Enterprise (“Consul”) such that operators with `operator:read` ACL permissions are able to read the Consul Connect CA configuration when explicitly configured with the `/v1/connect/ca/configuration` endpoint, including the private key. This allows the user to effectively privilege escalate by enabling the ability to mint certificates for any Consul Connect services. This would potentially allow them to masquerade (receive/send traffic) as any service in the mesh.

--

This PR increases the permissions required to read the Connect CA's private key when it was configured via the `/connect/ca/configuration` endpoint. They are now `operator:write`.
2020-11-19 10:14:48 -07:00
hashicorp-ci effe235562 auto-updated agent/uiserver/bindata_assetfs.go from commit 687ce1f9c 2020-11-19 16:13:04 +00:00
Daniel Nephin 671b8cf494
Merge pull request from hashicorp/dnephin/fix-multiple-http-listeners
agent: fix bug with multiple listeners
2020-11-18 16:52:29 -05:00
Daniel Nephin 79963be559 Use freeport
To prevent other tests which already use freeport from flaking when port 0 steals their reserved port.
2020-11-18 16:07:34 -05:00
hashicorp-ci b8659f77c4 auto-updated agent/uiserver/bindata_assetfs.go from commit 591a96d5b 2020-11-18 19:07:25 +00:00
hashicorp-ci cd003a14a5 auto-updated agent/uiserver/bindata_assetfs.go from commit 1edef424a 2020-11-18 19:00:19 +00:00
Daniel Nephin 738bf9efdc agent: fix bug with multiple listeners
Previously the listener was being passed to a closure in a loop without
capturing the loop variable. The result is only the last listener is
used, so the http/https servers only listen on one address.

This problem is fixed by capturing the variable by passing it into a
function.
2020-11-18 13:03:29 -05:00
hashicorp-ci b2605d90d2 auto-updated agent/uiserver/bindata_assetfs.go from commit 664f1d9aa 2020-11-18 11:17:06 +00:00
Kyle Havlovitz 9be7c6401c connect: update some function comments in CA manager 2020-11-17 16:00:19 -08:00
Daniel Nephin 3885835e8c acl: remove a test-only method 2020-11-17 18:16:34 -05:00
Daniel Nephin 0ee86935f0 Remove two unused delegate methods 2020-11-17 18:16:26 -05:00
Daniel Nephin 839429eb40
Merge pull request from hashicorp/dnephin/go-test-race-in-to-out-list
ci: change go-test-race package list to exclude list
2020-11-17 13:13:38 -05:00
Matt Keeler 66fd23d67f
Refactor to call non-voting servers read replicas ()
Co-authored-by: Kit Patella <kit@jepsen.io>
2020-11-17 10:53:57 -05:00
Kit Patella d15b6fddd3
Merge pull request from hashicorp/mkcp/telemetry/add-all-metric-definitions
Add metric definitions for all metrics known at Consul start
2020-11-16 15:54:50 -08:00
hashicorp-ci 56dbabf67b auto-updated agent/uiserver/bindata_assetfs.go from commit fe728855e 2020-11-16 23:41:31 +00:00
Freddy fe728855ed
Add DC and NS support for Envoy metrics ()
This PR updates the tags that we generate for Envoy stats.

Several of these come with breaking changes, since we can't keep two stats prefixes for a filter.
2020-11-16 16:37:19 -07:00
Kit Patella 8e554ee74b Merge branch 'mkcp/telemetry/add-all-metric-definitions' of ssh://github.com/hashicorp/consul into mkcp/telemetry/add-all-metric-definitions 2020-11-16 15:26:12 -08:00
Kit Patella ad4cebc1d8 fix some tests that were broken from the TelemetryConfig change 2020-11-16 15:22:36 -08:00
Kit Patella fc30f07cc7
linting: sort and group import 2020-11-16 14:17:24 -08:00
Kit Patella 2fe021f03c update runtime_test to handle PrometheusOpts expiry field change 2020-11-16 14:16:12 -08:00
Matt Keeler 748d56b8ab
Prevent panic if autopilot health is requested prior to leader establishment finishing. () 2020-11-16 17:08:17 -05:00
Kit Patella b81edac7bb use the MetricsPrefix to set the service name and provide as slice literal to avoid bugs from append modifying its first arg 2020-11-16 14:01:12 -08:00
Kit Patella 5e0e4098c9 push prometheus sink definiitons into prometheus.PrometheusOpts 2020-11-16 12:44:47 -08:00
Daniel Nephin b7367467f6
Merge pull request from hashicorp/dnephin/filtering-in-stream
stream: improve naming of Payload methods
2020-11-16 14:20:07 -05:00
Kit Patella 15af5ead0b trim help strings to save a few bytes 2020-11-16 11:02:11 -08:00
Kit Patella 3966ecb02f merge master 2020-11-16 10:46:53 -08:00
hashicorp-ci a54d1069b3 auto-updated agent/uiserver/bindata_assetfs.go from commit 959974e96 2020-11-16 15:27:40 +00:00
Kit Patella 5da2f1efa8 finish adding static server metrics 2020-11-13 16:26:08 -08:00
Kyle Havlovitz 16e95f1d7b Reorganize some CA manager code for correctness/readability 2020-11-13 14:46:01 -08:00
Kyle Havlovitz 6fba82a4fa connect: Add CAManager for synchronizing CA operations 2020-11-13 14:33:44 -08:00
Kyle Havlovitz af34b26221 connect: Add logic for updating secondary DC intermediate on config set 2020-11-13 14:33:44 -08:00
R.B. Boyer 9eb262252a
server: intentions CRUD requires connect to be enabled ()
Fixes 
2020-11-13 16:19:12 -06:00
Kit Patella 06d59c03b9 add the service name in the agent rather than in the definitions themselves 2020-11-13 13:18:04 -08:00
R.B. Boyer c7233ba871
server: remove config entry CAS in legacy intention API bridge code ()
Change so line-item intention edits via the API are handled via the state store instead of via CAS operations.

Fixes 
2020-11-13 14:42:21 -06:00
R.B. Boyer c52bc632df
server: skip deleted and deleting namespaces when migrating intentions to config entries () 2020-11-13 13:56:41 -06:00
Mike Morris 7af643ac37
ci: update to Go 1.15.4 and alpine:3.12 ()
* ci: stop building darwin/386 binaries

Go 1.15 drops support for 32-bit binaries on Darwin https://golang.org/doc/go1.15#darwin

* tls: ConnectionState::NegotiatedProtocolIsMutual is deprecated in Go 1.15, this value is always true

* correct error messages that changed slightly

* Completely regenerate some TLS test data

Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-11-13 13:02:59 -05:00
hashicorp-ci fe6b888269 auto-updated agent/uiserver/bindata_assetfs.go from commit 1059a51a3 2020-11-13 16:00:39 +00:00
hashicorp-ci 40cef22c17 auto-updated agent/uiserver/bindata_assetfs.go from commit 78b704be8 2020-11-13 15:44:14 +00:00
R.B. Boyer c003871c54
server: break up Intention.Apply monolithic method ()
The Intention.Apply RPC is quite large, so this PR attempts to break it down into smaller functions and dissolves the pre-config-entry approach to the breakdown as it only confused things.
2020-11-13 09:15:39 -06:00
Kit Patella 24a2471029 first pass on agent-configured prometheusDefs and adding defs for every consul metric 2020-11-12 18:12:12 -08:00
Daniel Nephin a397ec85eb
Merge pull request from hashicorp/dnephin/fix-grpc-metrics
grpc: fix metrics
2020-11-12 17:03:01 -05:00
hashicorp-ci 1cedf812e1 auto-updated agent/uiserver/bindata_assetfs.go from commit 6b2970402 2020-11-12 18:49:48 +00:00
R.B. Boyer 61eac21f1a
agent: return the default ACL policy to callers as a header ()
Header is: X-Consul-Default-ACL-Policy=<allow|deny>

This is of particular utility when fetching matching intentions, as the
fallthrough for a request that doesn't match any intentions is to
enforce using the default acl policy.
2020-11-12 10:38:32 -06:00
hashicorp-ci 4016918434 auto-updated agent/uiserver/bindata_assetfs.go from commit 7243f1f4f 2020-11-12 15:45:53 +00:00
Daniel Nephin b27457dac8 ci: go-test-race switch to exclude list
Most packages should pass the race detector. An exclude list ensures
that new packages are automatically tested with -race.

Also fix a couple small test races to allow more packages to be tested.

Returning readyCh requires a lock because it can be set to nil, and
setting it to nil will race without the lock.

Move the TestServer.Listening calls around so that they properly guard
setting TestServer.l. Otherwise it races.

Remove t.Parallel in a small package. The entire package tests run in a
few seconds, so t.Parallel does very little.

In auto-config, wait for the AutoConfig.run goroutine to stop before
calling readPersistedAutoConfig. Without this change there was a data
race on reading ac.config.
2020-11-11 14:44:57 -05:00
Daniel Nephin 1a137c29d6 grpc: fix grpc metrics
defaultMetrics was being set at package import time, which meant that it received an instance of
the original default. But lib/telemetry.InitTelemetry sets a new global when it is called.

This resulted in the metrics being sent nowhere.

This commit changes defaultMetrics to be a function, so it will return the global instance when
called. Since it is called after InitTelemetry it will return the correct metrics instance.
2020-11-11 14:27:25 -05:00
Matt Keeler 7ef9b04f90
Add a CLI command for retrieving the autopilot configuration. () 2020-11-11 13:19:02 -05:00
hashicorp-ci 848571a73a auto-updated agent/uiserver/bindata_assetfs.go from commit 6423a2c10 2020-11-11 17:03:36 +00:00
Matt Keeler 71da0209bf
Add a paramter in state store methods to indicate whether a resource insertion is from a snapshot restoration ()
The Catalog, Config Entry, KV and Session resources potentially re-validate the input as its coming in. We need to prevent snapshot restoration failures due to missing namespaces or namespaces that are being deleted in enterprise.
2020-11-11 11:21:42 -05:00
hashicorp-ci 37b1ab7f49 auto-updated agent/uiserver/bindata_assetfs.go from commit e1d977138 2020-11-11 14:48:38 +00:00
Daniel Nephin 3760e3d12d
Merge pull request from joel0/wrap-errors
Use error wrapping to preserve error type info
2020-11-10 18:27:08 -05:00
Daniel Nephin 45a9dd59b5
Merge pull request from joel0/wrap-eof
Wrap rpc error object
2020-11-10 17:04:11 -05:00
Joel May f600285eb4 Use error wrapping to preserve error type info 2020-11-10 21:50:09 +00:00
hashicorp-ci 77451d944e auto-updated agent/uiserver/bindata_assetfs.go from commit e18d8e299 2020-11-10 16:37:33 +00:00
hashicorp-ci 8f834c2d21 auto-updated agent/uiserver/bindata_assetfs.go from commit fb6202929 2020-11-10 14:42:02 +00:00
hashicorp-ci 031ab3f44f auto-updated agent/uiserver/bindata_assetfs.go from commit c8e40ee0d 2020-11-09 17:34:25 +00:00
Matt Keeler a3a653342b
Fix a bunch of linter warnings 2020-11-09 09:22:12 -05:00
Matt Keeler c048e86bb2
Switch to using the external autopilot module 2020-11-09 09:22:11 -05:00
hashicorp-ci 7a2c6dfd62 auto-updated agent/uiserver/bindata_assetfs.go from commit 5c0ec13fb 2020-11-09 09:31:52 +00:00
hashicorp-ci 908574058e auto-updated agent/uiserver/bindata_assetfs.go from commit d9672bca8 2020-11-09 09:19:52 +00:00
Mike Morris 75019baadd
chore: upgrade to gopsutil/v3 ()
* deps: update golang.org/x/sys

* deps: update imports to gopsutil/v3

* chore: make update-vendor
2020-11-06 20:48:38 -05:00
Daniel Nephin fb70c8bac2 stream: document that Payload must be immutable
If they are sent to EventPublisher.Publish.

Also document that PayloadEvents is expected to come from a subscription and that it is
not immutable.
2020-11-06 13:00:33 -05:00
R.B. Boyer 8baf158ea8
Revert "Add namespace support for metrics (OSS) ()" ()
This reverts commit 06b3b017d3.
2020-11-06 10:24:32 -06:00
hashicorp-ci cf537ac2f5 auto-updated agent/uiserver/bindata_assetfs.go from commit 3a68686cc 2020-11-06 15:04:29 +00:00
hashicorp-ci 24bc8451d5 auto-updated agent/uiserver/bindata_assetfs.go from commit 848f72f66 2020-11-06 09:31:18 +00:00
Freddy 06b3b017d3
Add namespace support for metrics (OSS) () 2020-11-05 18:24:29 -07:00
Daniel Nephin 43af0ba7a3 stream: rename FilterByKey 2020-11-05 19:21:16 -05:00
Daniel Nephin 868cfe1eac stream: Add HasReadPermission to Payload
Required now that filter is a method on PayloadEvents instead of Event
2020-11-05 19:17:18 -05:00
Daniel Nephin 36202f7938 stream: move event filtering to PayloadEvents
Removes the weirdness around PayloadEvents.FilterByKey
2020-11-05 17:50:17 -05:00
Daniel Nephin 79b5ca1ce6 stream: Remove unused method 2020-11-05 16:49:59 -05:00
R.B. Boyer 8e616a93c1
agent: sanitize ui metrics proxy header values on agent/self endpoint () 2020-11-05 13:25:27 -06:00
Daniel Nephin a33c50ef0d
Merge pull request from hashicorp/dnephin/backport-streaming-namespaces
streaming: backport namespace changes
2020-11-05 14:19:10 -05:00
Daniel Nephin c82f6ef2d8
Merge pull request from hashicorp/dnephin/event-fields
stream: support filtering by namespace
2020-11-05 14:18:35 -05:00
hashicorp-ci 977297390c auto-updated agent/uiserver/bindata_assetfs.go from commit 6ff094976 2020-11-05 19:12:03 +00:00
hashicorp-ci 9d15348565 auto-updated agent/uiserver/bindata_assetfs.go from commit 1ef18c4b6 2020-11-05 16:10:14 +00:00
R.B. Boyer 6ba776b4f3
agent: protect the ui metrics proxy endpoint behind ACLs ()
This ensures the metrics proxy endpoint is ACL protected behind a
wildcard `service:read` and `node:read` set of rules. For Consul
Enterprise these will need to span all namespaces:

```
service_prefix "" { policy = "read" }
node_prefix ""    { policy = "read" }

namespace_prefix "" {
  service_prefix "" { policy = "read" }
  node_prefix ""    { policy = "read" }
}
```

This PR contains just the backend changes. The frontend changes to
actually pass the consul token header to the proxy through the JS plugin
will come in another PR.
2020-11-04 12:50:03 -06:00
hashicorp-ci a2315bc839 auto-updated agent/uiserver/bindata_assetfs.go from commit 0f6c0a5c1 2020-11-04 09:37:51 +00:00
hashicorp-ci 1a5d4cfe43 auto-updated agent/uiserver/bindata_assetfs.go from commit d5d4155e1 2020-11-03 14:14:58 +00:00
hashicorp-ci 738ff1801f auto-updated agent/uiserver/bindata_assetfs.go from commit 56c2ff56e 2020-11-02 18:43:31 +00:00
hashicorp-ci c28f489a9a auto-updated agent/uiserver/bindata_assetfs.go from commit bf32a1799 2020-11-02 16:11:45 +00:00
hashicorp-ci 907c4ad789 auto-updated agent/uiserver/bindata_assetfs.go from commit 314eeda95 2020-11-02 14:40:27 +00:00
R.B. Boyer a66c4591d7
agent: introduce path allow list for requests going through the metrics proxy ()
Added a new option `ui_config.metrics_proxy.path_allowlist`. This defaults to `["/api/v1/query", "/api/v1/query_range"]` when the metrics provider is set to `prometheus`.

Requests that do not use one of the allow-listed paths (via exact match) get a 403 Forbidden response instead.
2020-10-30 16:49:54 -05:00
Daniel Nephin b532e092dc structs: add a namespace test for CheckServiceNode.CanRead 2020-10-30 15:07:04 -04:00
Daniel Nephin 9b2fae9bac cache-type: use namespace in tests
to verify that the namespace is passed through correctly to the server.
2020-10-30 15:07:04 -04:00
Daniel Nephin b95b14e168 state: test EventPayloadCheckServiceNode.FilterByKey
Also fix a bug in that function when only one of key or namespace were the empty string.
2020-10-30 14:35:57 -04:00
Daniel Nephin 56d6079da3 stream: Add tests for filterByKey with namespace
And fix a bug where a request with a Namespace but no Key would not be properly filtered
2020-10-30 14:35:42 -04:00
Daniel Nephin 2c00045161 stream: Move FilterByKey events to a table
In preparation for adding new tests.
2020-10-30 14:35:28 -04:00
Daniel Nephin 43c5803a25 state: use enterprise meta for creating events 2020-10-30 14:34:04 -04:00
Daniel Nephin 0ad2406d7c stream: include the namespace in the snap cache key
Otherwise the wrong snapshot could be returned when the same key is used in different namespaces
2020-10-30 14:34:04 -04:00
Daniel Nephin c42fe5ae43 subscribe: set the request namespace 2020-10-30 14:34:04 -04:00
hashicorp-ci b3bf1229ac auto-updated agent/uiserver/bindata_assetfs.go from commit cf2cfbaf2 2020-10-30 15:27:01 +00:00
R.B. Boyer fa4b0854fb
state: ensure we unblock intentions queries upon the upgrade to config entries ()
1. do a state store query to list intentions as the agent would do over in `agent/proxycfg` backing `agent/xds`
2. upgrade the database and do a fresh `service-intentions` config entry write
3. the blocking query inside of the agent cache in (1) doesn't notice (2)
2020-10-29 15:28:31 -05:00
R.B. Boyer b24b4169e1 restore prior signature of test helper so enterprise compiles 2020-10-29 13:52:15 -05:00
hashicorp-ci 01dbf43fb1 auto-updated agent/uiserver/bindata_assetfs.go from commit 1d6961248 2020-10-29 18:33:41 +00:00
Michael Montgomery 5b6ac035ff Resolves . Adds new option to configure HTTP Server's MaxHeaderBytes with option `-http-max-header-bytes`
Adds tests for behavior
2020-10-29 12:38:19 -05:00
Daniel Nephin a5dd2001cf stream: remove Event.Key
Makes Payload a type with FilterByKey so that Payloads can implement
filtering by key. With this approach we don't need to expose a Namespace
field on Event, and we don't need to invest micro formats or require a
bunch of code to be aware of exactly how the key field is encoded.
2020-10-28 16:48:04 -04:00
Daniel Nephin 1c094da40d state: use go-cmp for comparison
The output of the previous assertions made it impossible to debug the tests without code changes.

With go-cmp comparing the entire slice we can see the full diffs making it easier to debug failures.
2020-10-28 16:33:00 -04:00
Daniel Nephin 68342a0cb5 proto: remove Event.Key field
The field is never used, and the value is available from the payload.
2020-10-28 16:33:00 -04:00
Daniel Nephin 9a1e845be8 proto: remove Event.Namespace field
All events are part of a single Topic, so we don't need this field.
2020-10-28 16:33:00 -04:00
Daniel Nephin 3dfb7c224b stream: Use a no-op event publisher if streaming is disabled 2020-10-28 13:54:19 -04:00
Daniel Nephin 23eee604c9 store: use a ReadDB for snapshots
to remove the cyclic dependency between the snapshot handlers and the state.Store
2020-10-28 13:07:42 -04:00
Daniel Nephin 7b9ee25956
Merge pull request from hashicorp/dnephin/streaming-without-cache-query-param
streaming: rename config and remove requirement for cache=1
2020-10-28 12:33:25 -04:00
Daniel Nephin 477d665309
Merge pull request from hashicorp/dnephin/remove-txn-readtxn
state: Use ReadTxn everywhere
2020-10-28 12:32:47 -04:00
Kyle Havlovitz 9f893307de
Merge pull request from hashicorp/vault-token-lookupself
connect: Use the lookup-self endpoint for Vault token
2020-10-27 14:34:03 -07:00
Daniel Nephin f0ac093fef agent/grpc: add connection count metrics
Gauge metrics are great for understanding the current state, but can somtimes hide problems
if there are many disconnect/reconnects.

This commit adds counter metrics for connections and streams to make it easier to see the
count of newly created connections and streams.
2020-10-27 16:49:49 -04:00
Daniel Nephin 5319ba02b0 agent/grpc: rename metrics
These new names should make it easier to add counter metics with similar prefixes
2020-10-27 16:49:49 -04:00
Daniel Nephin c82d6aa4ff
Merge pull request from hashicorp/dnephin/grpc-resolve-node-id
agent/grpc: fix some test flakes and handle duplicate server IDs in the pool
2020-10-27 16:47:37 -04:00
Daniel Nephin 74ac34e358
Merge pull request from hashicorp/dnephin/lib-ttlcache
lib/ttlcache: extract a new package from agent/cache
2020-10-27 16:43:10 -04:00
Daniel Nephin 0f81915495
Merge pull request from hashicorp/dnephin/stream-filter
streaming: apply filter to a single item
2020-10-27 16:39:43 -04:00
Daniel Nephin 8bcd5040c7 agent/grpc: Add an integration test for ClientPool with TLS
Also deregister the resolver.Builder in tests.
2020-10-27 16:34:18 -04:00
Daniel Nephin 8a785a351c agent/grpc: pass metrics to constructor
Instead of referencing a package var. This does not fix the flaky test, but it seems more correct.
2020-10-27 16:34:17 -04:00
Daniel Nephin d19657404f agent/grpc: fix a flaky test by performing more retries
Instead of using retry.Run, which appears to have problems in some cases where it does not
emit an error message, use a for loop.

Increase the number of attempts and remove any sleep, since this operation is not that expensive to do
in a tight loop
2020-10-27 16:34:17 -04:00
Daniel Nephin df405ac978 agent/grpc: remove misleading warnings from test output
Handle shutdown properly in tests so that the tests don't warn about using a closed connection.
2020-10-27 16:34:16 -04:00
Daniel Nephin e101aa8a74 agent/grpc: fix a flake in TestHandler_EmitsStats 2020-10-27 16:34:16 -04:00
Daniel Nephin 19da9c3a9b agent/grpc: use a separate channel for closing the Accept
Closing l.conns can lead to a race and a 'panic: send on closed chan' when a
connection is in the middle of being handled when the server is shutting down.

Found using '-race -count=800'
2020-10-27 16:34:15 -04:00
Daniel Nephin d8299670cc agent/grpc/resolver: namespace the server ID with the DC name
So that if two datacenters end up with overlapping serverIDs we don't send requests to the wrong server
2020-10-27 16:34:15 -04:00
Kyle Havlovitz f700a5707b connect: Use the lookup-self endpoint for Vault token 2020-10-27 13:03:45 -07:00
hashicorp-ci bea3d0fd96 auto-updated agent/uiserver/bindata_assetfs.go from commit f4208b5fb 2020-10-27 14:56:48 +00:00
hashicorp-ci 7a1538f747 auto-updated agent/uiserver/bindata_assetfs.go from commit 30da884d5 2020-10-27 14:31:16 +00:00
hashicorp-ci b6113eba09 auto-updated agent/uiserver/bindata_assetfs.go from commit ed6a2c150 2020-10-27 14:09:17 +00:00
hashicorp-ci 6c39fcecf3 auto-updated agent/uiserver/bindata_assetfs.go from commit 827e53694 2020-10-27 13:36:44 +00:00
hashicorp-ci 5460744a95 auto-updated agent/uiserver/bindata_assetfs.go from commit 52d7283cd 2020-10-26 19:56:11 +00:00
Paul Banks 52d7283cd6
UI metrics provider dc ()
* Plumb Datacenter and Namespace to metrics provider in preparation for them being usable.

* Move metrics loader/status to a new component and show reason for being disabled.

* Remove stray console.log

* Rebuild AssetFS to resolve conflicts

* Yarn upgrade

* mend
2020-10-26 19:48:23 +00:00
hashicorp-ci f585db82cf auto-updated agent/uiserver/bindata_assetfs.go from commit c4f027fa0 2020-10-26 16:55:52 +00:00
Daniel Nephin bd44952c2e streaming: disable streaming when requesting connect events
Until the correct events are created for terminating gateways.
2020-10-26 11:55:49 -04:00
Daniel Nephin c398a6b272 state: disable streaming connect topic 2020-10-26 11:49:47 -04:00
hashicorp-ci 9772c12426 auto-updated agent/uiserver/bindata_assetfs.go from commit 948917c6b 2020-10-26 09:34:58 +00:00
Daniel Nephin 853667e7d8 health: change the name of UseStreamingBackend config
Remove it from the cache section, and update the docs.
2020-10-23 17:47:01 -04:00
Daniel Nephin a62dcc9bfe health: use streaming, even when cache=1 is not set 2020-10-23 17:39:55 -04:00
R.B. Boyer 58387fef0a
server: config entry replication now correctly uses namespaces in comparisons ()
Previously config entries sharing a kind & name but in different
namespaces could occasionally cause "stuck states" in replication
because the namespace fields were ignored during the differential
comparison phase.

Example:

Two config entries written to the primary:

    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo

Under the covers these both get saved to memdb, so they are sorted by
all 3 components (kind,name,namespace) during natural iteration. This
means that before the replication code does it's own incomplete sort,
the underlying data IS sorted by namespace ascending (bar comes before
foo).

After one pass of replication the primary and secondary datacenters have
the same set of config entries present. If
"kind=A,name=web,namespace=bar" were to be deleted, then things get
weird. Before replication the two sides look like:

primary: [
    kind=A,name=web,namespace=foo
]
secondary: [
    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo
]

The differential comparison phase walks these two lists in sorted order
and first compares "kind=A,name=web,namespace=foo" vs
"kind=A,name=web,namespace=bar" and falsely determines they are the SAME
and are thus cause an update of "kind=A,name=web,namespace=foo". Then it
compares "<nothing>" with "kind=A,name=web,namespace=foo" and falsely
determines that the latter should be DELETED.

During reconciliation the deletes are processed before updates, and so
for a brief moment in the secondary "kind=A,name=web,namespace=foo" is
erroneously deleted and then immediately restored.

Unfortunately after this replication phase the final state is identical
to the initial state, so when it loops around again (rate limited) it
repeats the same set of operations indefinitely.
2020-10-23 13:41:54 -05:00
Daniel Nephin 0f1fb24d19 state: convert the remaining functions to ReadTxn
Required also converting some of the transaction functions to WriteTxn
because TxnRO() called the same helper as TxnRW.

This change allows us to return a memdb.Txn for read-only txn instead of
wrapping them with state.txn.
2020-10-23 14:29:22 -04:00
Daniel Nephin 8bd1a2cd16
Merge pull request from hashicorp/dnephin/stream-close-on-unsub
stream: close the subscription on Unsubscribe
2020-10-23 12:58:12 -04:00
Freddy 9c04cbc40f
Add HasExact to topology endpoint () 2020-10-23 10:45:41 -06:00
hashicorp-ci 46071cbb03 auto-updated agent/uiserver/bindata_assetfs.go from commit eedee07e7 2020-10-23 16:32:06 +00:00
hashicorp-ci e236a58dba auto-updated agent/uiserver/bindata_assetfs.go from commit 7559f64d0 2020-10-23 14:45:43 +00:00
hashicorp-ci 74b8c024e8 auto-updated agent/uiserver/bindata_assetfs.go from commit 2a8a80638 2020-10-23 13:50:57 +00:00
hashicorp-ci a751688bc9 auto-updated agent/uiserver/bindata_assetfs.go from commit 45554e4e9 2020-10-23 08:32:18 +00:00
R.B. Boyer a2c50d3303
connect: add support for envoy 1.16.0, drop support for 1.12.x, and bump point releases as well ()
Supported versions will be: "1.16.0", "1.15.2", "1.14.5", "1.13.6"
2020-10-22 13:46:19 -05:00
Daniel Nephin 3c52f3854b
Merge pull request from hashicorp/dnephin/resolver-balance
agent/router: refactor calculation of delay between rebalances.
2020-10-22 13:52:10 -04:00
Daniel Nephin fb57d9b26a stream: close the subscription on Unsubscribe 2020-10-22 13:39:27 -04:00
Daniel Nephin 3a55c30a05
Merge pull request from ShimmerGlass/fix-sidecar-deregister-after-restart
Fix: service LocallyRegisteredAsSidecar property is not persisted
2020-10-22 13:26:55 -04:00
Daniel Nephin d4ae586ccb
Merge pull request from hashicorp/dnephin/dns-small-cleanup-1
dns: remove goto INVALID and a naked return
2020-10-21 18:41:11 -04:00
Daniel Nephin 733ce9ad4a
Merge pull request from hashicorp/dnephin/resolver-balance-test
agent/router: improve  the test for refreshServerRebalanceTimer
2020-10-21 18:40:32 -04:00
Daniel Nephin 8b601fdcac
Merge pull request from amenzhinsky/fix-grpc-use-tls-mapping
Fix GRPCUseTLS flag HTTP API mapping
2020-10-21 18:37:11 -04:00
Daniel Nephin 9b1944dae0 dns: remove goto INVALID and a naked return
This commit is one small step toward modernizing and making the DNSServer readable.
2020-10-21 15:16:03 -04:00
Daniel Nephin 0d4fa882b3 lib/ttlcache: unexport key and additional godoc 2020-10-20 19:16:03 -04:00
Daniel Nephin c17baadbf8 lib/ttlcache: add a constant for NotIndexed 2020-10-20 19:10:20 -04:00
Daniel Nephin 6c09ab3dd8 cache: fix a bug with Prepopulate
Prepopulate was setting entry.Expiry.HeapIndex to 0. Previously this would result in a call to heap.Fix(0)
which wasn't correct, but was also not really a problem because at worse it would re-notify.

With the recent change to extract cachettl it was changed to call Update(idx), which would have updated
the wrong entry.

A previous commit removed the setting of entry.Expiry so that the HeapIndex would be reported
as -1, and this commit adds a test and handles the -1 heap index.
2020-10-20 19:10:20 -04:00
Daniel Nephin bbb816aa8a lib/ttlcache: extract package from agent/cache 2020-10-20 19:10:20 -04:00
Daniel Nephin c4122edd22 cache: export ExpiryHeap
and hide internal methods on an unexported type, so that when it is extrated those methods are not exported.
2020-10-20 19:10:20 -04:00
Daniel Nephin 343d133183 cache: Refactor heap.notify to make it more explicit.
And remove duplicate notifications.

Instead of performing the check in the heap implementation, check the
index in the higher level interface (Add,Remove,Update) and notify if one
of the relevant indexes is 0.
2020-10-20 19:10:20 -04:00
Daniel Nephin 499f2822cf cache: Move more of the expiryLoop into the Heap 2020-10-20 19:10:20 -04:00
Daniel Nephin 2cdc90e01b cache: extract cache eviction heap
Start creating an interface that doesn't require using heap and hides more of the
entry internals.
2020-10-20 19:10:19 -04:00
Daniel Nephin 3c8929c7e1 streaming: apply filter to a single item
Instead of the whole map. This should save a lot of time performing reflecting on a large map.
The filter does not change, so there is no reason to re-apply it to older entries.
2020-10-19 18:24:02 -04:00
hashicorp-ci 6b68e1c997 auto-updated agent/uiserver/bindata_assetfs.go from commit 2773daedc 2020-10-19 16:35:48 +00:00
hashicorp-ci 13c06ba8bc auto-updated agent/uiserver/bindata_assetfs.go from commit b24c2227d 2020-10-19 16:19:40 +00:00
hashicorp-ci 6513faf14c auto-updated agent/uiserver/bindata_assetfs.go from commit 02dcd422f 2020-10-19 14:50:52 +00:00
hashicorp-ci 8ff996145a auto-updated agent/uiserver/bindata_assetfs.go from commit bf8bb7415 2020-10-19 14:25:54 +00:00
Joel May 8c6d6648cd Wrap rpc error object 2020-10-16 00:42:05 +00:00
Daniel Nephin 0003720f78 agent/router: refactor calculation of delay between rebalances.
This change attempts to make the delay logic more obvious by:

* remove indirection, inline a bunch of function calls
* move all the code and constants next to each other
* replace the two constant values with a single value
* reword the comments.
2020-10-15 15:59:36 -04:00
Daniel Nephin 119c446cf2 agent/router: Add bounds test cases 2020-10-15 14:43:29 -04:00
Daniel Nephin 12e174900b router: organize the test by number of servers
And adddd some additional cases to show where the minimum value stops being used
2020-10-15 13:53:37 -04:00
Daniel Nephin 8697cc2b45 router: make refreshServerRebalanceTimer test a lot more strict 2020-10-15 12:05:07 -04:00
hashicorp-ci 40c16f6ee4 auto-updated agent/uiserver/bindata_assetfs.go from commit d08e13e5b 2020-10-15 14:49:01 +00:00
hashicorp-ci 55c89133ba auto-updated agent/uiserver/bindata_assetfs.go from commit 5fd79ba40 2020-10-14 08:07:29 +00:00
Mathilde Gilles 1c8369b3c3 Fix: service LocallyRegisteredAsSidecar property is not persisted
When a service is deregistered, we check whever matching services were
registered as sidecar along with it and deregister them as well.
To determine if a service is indeed a sidecar we check the
structs.ServiceNode.LocallyRegisteredAsSidecar property. However, to
avoid interal API leakage, it is excluded from JSON serialization,
meaning it is not persisted to disk either.
When the agent is restarted, this property lost and sidecars are no
longer deregistered along with their parent service.
To fix this, we now specifically save this property in the persisted
service file.
2020-10-13 19:38:58 +02:00
hashicorp-ci 5ede07e0e7 auto-updated agent/uiserver/bindata_assetfs.go from commit 7afcd7755 2020-10-13 16:43:31 +00:00
Mike Morris 9455805ceb chore: regenerate bindata_assetfs 2020-10-12 15:44:33 -04:00
Mike Morris f3f6e57ed6 Merge branch 'master' into release/1.9.0-beta1 2020-10-12 15:32:58 -04:00
hashicorp-ci 700c163ed3
update bindata_assetfs.go 2020-10-12 18:30:30 +00:00
hashicorp-ci 564ab4146e auto-updated agent/uiserver/bindata_assetfs.go from commit 868cd47a6 2020-10-12 16:28:44 +00:00
Pierre Souchay 9b7ed75552
Consul Service meta wrongly computes and exposes non_voter meta ()
* Consul Service meta wrongly computes and exposes non_voter meta

In Serf Tags, entreprise members being non-voters use the tag
`nonvoter=1`, not `non_voter = false`, so non-voters in members
were wrongly displayed as voter.

Demonstration:

```
consul members -detailed|grep voter
consul20-hk5 10.200.100.110:8301   alive   acls=1,build=1.8.4+ent,dc=hk5,expect=3,ft_fs=1,ft_ns=1,id=xxxxxxxx-5629-08f2-3a79-10a1ab3849d5,nonvoter=1,port=8300,raft_vsn=3,role=consul,segment=<all>,use_tls=1,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302
```

* Added changelog

* Added changelog entry
2020-10-09 17:18:24 -04:00
hashicorp-ci b73dea7942 auto-updated agent/uiserver/bindata_assetfs.go from commit 851705e93 2020-10-09 21:01:02 +00:00
hashicorp-ci 4f00a02737 auto-updated agent/uiserver/bindata_assetfs.go from commit 27048a061 2020-10-09 20:35:59 +00:00
Paul Banks 27048a0612
Add metrics rendering to the new topology view. ()
* Remove unused StatsCard component

* Create Card and Stats contextual components with styling

* Send endpoint, item, and protocol to Stats as props

* WIP basic plumbing for metrics in Ember

* WIP metrics data source now works for different protocols and produces reasonable mock responses

* WIP sparkline component

* Mostly working metrics and graphs in topology

* Fix date in tooltip to actually be correct

* Clean up console.log

* Add loading frame and create a style sheet for Stats

* Various polish fixes:

 - Loading state for graph
 - Added fake latency cookie value to test loading
 - If metrics provider has no series/stats for the service show something that doesn't look broken
 - Graph hover works right to the edge now
 - Stats boxes now wrap so they are either shown or not as will fit not cut off
 - Graph resizes when browser window size changes
 - Some tweaks to number formats and stat metrics to make them more compact/useful

* Thread Protocol through topology model correctly

* Rebuild assetfs

* Fix failing tests and remove stats-card now it's changed and become different

* Fix merge conflict

* Update api doublt

* more merge fixes

* Add data-permission and id attr to Card

* Run JS linter

* Move things around so the tests run with everything available

* Get tests passing:

1. Remove fakeLatency setTimeout (will be replaced with CONSUL_LATENCY
in mocks)
2. Make sure any event handlers are removed

* Make sure the Consul/scripts are available before the app

* Make sure interval gets set if there is no cookie value

* Upgrade mocks so we can use CONSUL_LATENCY

* Fix handling of no series values from Prometheus

* Update assetfs and fix a comment

* Rebase and rebuild assetfs; fix tcp metric series units to be bits not bytes

* Rebuild assetfs

* Hide stats when provider is not configured

Co-authored-by: kenia <keniavalladarez@gmail.com>
Co-authored-by: John Cowen <jcowen@hashicorp.com>
2020-10-09 21:31:15 +01:00
s-christoff 9bb348c6c7
Enhance the output of consul snapshot inspect () 2020-10-09 14:57:29 -05:00
Kyle Havlovitz ff12fc9f38 Stop intermediate renew routine on leader stop 2020-10-09 12:30:57 -07:00
Kyle Havlovitz e5ab1b45bc
Merge pull request from hashicorp/renew-intermediate-primary
connect: Enable renewing the intermediate cert in the primary DC
2020-10-09 12:18:59 -07:00
hashicorp-ci 45c194f21b auto-updated agent/uiserver/bindata_assetfs.go from commit 766b28b8e 2020-10-09 18:44:15 +00:00
Daniel Nephin ea77eccb14
Merge pull request from hashicorp/streaming/add-config
streaming: add config and docs
2020-10-09 14:33:58 -04:00
Daniel Nephin e7d505dc33 config: add field for enabling streaming in the client
agent: register the new streaming cache-type
2020-10-09 14:11:34 -04:00
Daniel Nephin 0d653b184b
Merge pull request from hashicorp/dnephin/add-steps-to-subscribe-tests
subscribe: add steps to long test cases, and add new cases for converting Events
2020-10-09 13:54:59 -04:00
Daniel Nephin 8e8ba6301d
Merge pull request from hashicorp/dnephin/go-test-race-more-pkgs
ci: go test -race more packages
2020-10-09 13:48:56 -04:00
Chris Piraino 30540e406b
Emit service usage metrics with correct labeling strategy ()
Previously, we would emit service usage metrics both with and without a
namespace label attached. This is problematic in the case when you want
to aggregate metrics together, i.e. "sum(consul.state.services)". This
would cause services to be counted twice in that aggregate, once via the
metric emitted with a namespace label, and once in the metric emited
without any namespace label.
2020-10-09 11:01:45 -05:00
Kyle Havlovitz 876500e0dc Fix intermediate refresh test comments 2020-10-09 08:53:33 -07:00
Matt Keeler 0d8bf9c62c
Remove oss/ent divergence of runtime_test.go () 2020-10-09 10:45:11 -04:00
Matt Keeler 8f890bc027
Add capability for the v1/connect/ca/roots endpoint to return a PEM encoded certificate chain ()
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-09 10:43:33 -04:00
R.B. Boyer e113dc0fe2
upstream some differences from enterprise () 2020-10-09 09:42:53 -05:00
hashicorp-ci 19abd50412 auto-updated agent/uiserver/bindata_assetfs.go from commit b6f686fec 2020-10-09 13:37:35 +00:00
R.B. Boyer b6f686fecb
uiserver: upstream refactors done elsewhere () 2020-10-09 08:32:39 -05:00
Kyle Havlovitz 01ce9f5b18 Update CI for leader renew CA test using Vault 2020-10-09 05:48:15 -07:00
Kyle Havlovitz 4fc0f6d9a4
Merge branch 'master' into renew-intermediate-primary 2020-10-09 04:40:34 -07:00
Kyle Havlovitz e13f4af06b connect: Check for expired root cert when cross-signing 2020-10-09 04:35:56 -07:00
Paul Banks f11b759ddf
Actually proxy the query string too 2020-10-09 12:27:24 +01:00
Daniel Nephin 3ff6c5b3d3 cache-types: skip tests with races 2020-10-08 20:15:13 -04:00
Daniel Nephin af8a617797 grpc: fix data rate in stats handler test 2020-10-08 19:43:49 -04:00
Freddy 13df5d5bf8
Add protocol to the topology endpoint response () 2020-10-08 17:31:54 -06:00
Daniel Nephin c5d57c9f07 subscribe: add test cases for newEventFromStreamEvent 2020-10-08 18:48:17 -04:00
Kit Patella adeabf2399
Merge pull request from hashicorp/mkcp/telemetry/consul.api.http
Add flag for disabling 1.9 metrics backwards compatibility and warnings when set to default
2020-10-08 13:22:37 -07:00
R.B. Boyer 46a4ef7248
agent: allow the /v1/connect/intentions/match endpoint to use the agent cache ()
This is the recommended proxy integration API for listing intentions
which should not require an active connection to the servers to resolve
after the initial cache filling.
2020-10-08 14:51:53 -05:00
Daniel Nephin f185124320 subscribe: Add steps to rpc/subscribe tests
To make them easier to follow
2020-10-08 15:38:01 -04:00
hashicorp-ci de99c78f14 auto-updated agent/uiserver/bindata_assetfs.go from commit 13dfde75a 2020-10-08 19:36:48 +00:00
Matt Keeler 38f5ddce2a
Add per-agent reconnect timeouts ()
This allows for client agent to be run in a more stateless manner where they may be abruptly terminated and not expected to come back. If advertising a per-agent reconnect timeout using the advertise_reconnect_timeout configuration when that agent leaves, other agents will wait only that amount of time for the agent to come back before reaping it.

This has the advantageous side effect of causing servers to deregister the node/services/checks for that agent sooner than if the global reconnect_timeout was used.
2020-10-08 15:02:19 -04:00
Paul Banks 332477f24c
Add /v1/internal/ui/metrics-proxy API endpoint that proxies to a configured metrics provider backend. 2020-10-08 17:32:29 +01:00
Paul Banks 80910b46f3
Fix merge conflicts 2020-10-08 17:31:36 +01:00
Daniel Nephin b93577c94f config: add field for enabling streaming RPC endpoint 2020-10-08 12:11:20 -04:00
Daniel Nephin b5b790d4c0 streaming: Use a shorter LastGetTTL for the cache 2020-10-08 12:11:20 -04:00
hashicorp-ci fc6a7df5f6 auto-updated agent/uiserver/bindata_assetfs.go from commit b373456c7 2020-10-08 15:56:05 +00:00
Freddy 164ce57db2
Support ingress gateways in mesh viz endpoint ()
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-08 09:47:09 -06:00
hashicorp-ci 418fe6ba32 auto-updated agent/uiserver/bindata_assetfs.go from commit ef7b1f8a4 2020-10-08 15:41:40 +00:00
hashicorp-ci 12e735dbca auto-updated agent/uiserver/bindata_assetfs.go from commit d849f025c 2020-10-08 15:08:16 +00:00
Daniel Nephin fcaf362c9f
Merge pull request from hashicorp/streaming/materialize-view-fix-service-unique-id
streaming: Use an ID that includes namespace to store services in the materialized view
2020-10-07 21:28:53 -04:00
Daniel Nephin 2513f42c68
Merge pull request from hashicorp/streaming/materialize-view
Add StreamingHealthServices cache-type
2020-10-07 21:26:38 -04:00
Daniel Nephin b103568e98
Merge pull request from hashicorp/streaming/add-subscribe-service-batch-events
stream: handle batch events as a special case of Event
2020-10-07 21:25:32 -04:00
Daniel Nephin da6400192b
Merge pull request from hashicorp/streaming/add-subscribe-service
subscribe: add subscribe service for streaming change events
2020-10-07 21:24:03 -04:00
Freddy da91e999f6
Return intention info in svc topology endpoint () 2020-10-07 18:35:34 -06:00
Kit Patella 7fe2f80b4b add config flag to disable 1.9 metrics backwards compatibility. Add warnings on start and reload on default value 2020-10-07 17:12:52 -07:00
Daniel Nephin 21c21191f4 structs: add CheckServiceNode.CanRead
And use it from the subscribe endpoint.
2020-10-07 18:15:13 -04:00
hashicorp-ci 8655b24639 auto-updated agent/uiserver/bindata_assetfs.go from commit 0d0de4309 2020-10-07 14:13:29 +00:00
hashicorp-ci eb4bc23a03 auto-updated agent/uiserver/bindata_assetfs.go from commit a61b6c53d 2020-10-07 13:50:39 +00:00
hashicorp-ci aa0f5ff839 auto-updated agent/uiserver/bindata_assetfs.go from commit 222555c4c 2020-10-07 08:09:54 +00:00
R.B. Boyer 1b413b0444
connect: support defining intentions using layer 7 criteria ()
Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).
2020-10-06 17:09:13 -05:00
Daniel Nephin 8b887af0d3 streaming: store services with a unique ID that includes namespace 2020-10-06 16:54:56 -04:00
R.B. Boyer a2a8e9c783
connect: intentions are now managed as a new config entry kind "service-intentions" ()
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
Daniel Nephin 5972bdc87c streaming: improve godoc for cache-type
And fix a bug where any error that implemented the temporary interface was considered
a temporary error, even when the method would return false.
2020-10-06 13:52:02 -04:00
Daniel Nephin d0b87cd5d0 submatview: remove notifyUpdateLocked from reset
This call appears to only be necessary because reset() was called from
NewMaterializer.

This commit has the constructor set a default value for updateCh, and
removes both the call to reset() from New(), and the call to
notifyUpdateLocked() from reset().

This should ensure that we do not notify the Fetch() call before we have new
values to report.
2020-10-06 13:22:02 -04:00
Daniel Nephin 5eab1d8cef submatview: remove method receiver from handlers 2020-10-06 13:22:02 -04:00
Daniel Nephin 3fa08beecf submatview: add a test for handling of NewSnapshotToFollow
Also add some godoc
Rename some vars and functions
Fix a data race in the new cache test for entry closing.
2020-10-06 13:22:02 -04:00
Daniel Nephin 534d8b45bb submatview: refactor Materializer
Refactor of Materializer.Run
Use handlers to manage state in Materializer
Rename Materializer receiver
rename m.l to m.lock, and flip some conditionals to remove the negative.
Improve godoc, rename Deps, move resetErr, and pass err into notifyUpdate
Update for NewSnapshotToFollow events
Refactor to move context cancel out of Materializer
2020-10-06 13:22:02 -04:00
Daniel Nephin e849f6d7ac submatview: Move the 'use materialize from result.State' logic
No need to do all this other work if we have one already.

This logic moved closer to this call site 3 times during the process
of refactoring.
2020-10-06 13:22:02 -04:00
Daniel Nephin edf30b2714 submatview: Move Materializer to submatview package 2020-10-06 13:22:02 -04:00
Daniel Nephin ed45957ffb submatview: Refactor MaterializeView
Replace InitFilter with Reset.
Removes the need to store a fatalErr and the cache-type, and removes the need to recreate the filter
each time.
Pass dependencies into MaterializedView.
Remove context from MaterializedView.
Rename state to view.
Rename MaterialziedView to Materialzier.
Rename to NewMaterializer
Pass in retry.Waiter
2020-10-06 13:22:02 -04:00
Daniel Nephin b576a2d3c7 cache-types: Update Streaming health cache-type
To use latest protobuf types
2020-10-06 13:22:02 -04:00
Daniel Nephin 132b76acef agent/cache: Add cache-type and materialized view for streaming health
Extracted from d97412ce4c

Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 13:21:57 -04:00
Daniel Nephin b27068b72a stream: Return a single event from a subscription.Next
Handle batch events as a single event
2020-10-06 13:18:20 -04:00
Daniel Nephin f5d11562f2 subscribe: update to use NewSnapshotToFollow event 2020-10-06 12:49:35 -04:00
Daniel Nephin e3290f5971 Move agent/subscribe -> agent/rpc/subscribe 2020-10-06 12:49:35 -04:00
Daniel Nephin dbb8bd679f subscirbe: extract streamID and logging from Subscribe
By extracting all of the tracing logic the core logic of the Subscribe
endpoint is much easier to read.
2020-10-06 12:49:35 -04:00
Daniel Nephin 9e4ebacb05 subscribe: add integration test for acl token updates 2020-10-06 12:49:35 -04:00
Daniel Nephin 39beed0af6 subscribe: add integration test for filtering events by acl 2020-10-06 12:49:35 -04:00
Daniel Nephin 083f4e8f57 subscribe: Add an integration test for forward to DC 2020-10-06 12:49:35 -04:00
Daniel Nephin 013ababda4 subscribe: add first integration test for Server 2020-10-06 12:49:35 -04:00
Daniel Nephin d0256a0c07 subscribe: add a stateless subscribe service for the gRPC server
With a Backend that provides access to the necessary dependencies.
2020-10-06 12:49:35 -04:00
Daniel Nephin f4ea3066fb subscribe: add commented out test cases
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin 106d781dc9 subscribe: add initial impl from streaming-rpc-final branch
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin 364f6589c8
Merge pull request from hashicorp/streaming/rename-framing-events
stream: remove EndOfEmptySnapshot, add NewSnapshotToFollow
2020-10-06 12:42:58 -04:00
Daniel Nephin a5c50c982d
Merge pull request from hashicorp/dnephin/service-health-interface
rpcclient: Add health.Client and use it in http and dns
2020-10-06 12:13:09 -04:00
R.B. Boyer 4998a08c56
server: create new memdb table for storing system metadata ()
This adds a new very tiny memdb table and corresponding raft operation
for updating a very small effective map[string]string collection of
"system metadata". This can persistently record a fact about the Consul
state machine itself.

The first use of this feature will come in a later PR.
2020-10-06 10:08:37 -05:00
hashicorp-ci 356998d91e auto-updated agent/uiserver/bindata_assetfs.go from commit a7038b48c 2020-10-06 13:43:27 +00:00
hashicorp-ci bc5ca8b638 auto-updated agent/uiserver/bindata_assetfs.go from commit 5a39be47d 2020-10-06 13:32:27 +00:00
hashicorp-ci 36d219e2cc auto-updated agent/uiserver/bindata_assetfs.go from commit 76d3909f3 2020-10-06 08:42:39 +00:00
hashicorp-ci 4c9630ffad auto-updated agent/uiserver/bindata_assetfs.go from commit e4a0dcf10 2020-10-06 08:36:18 +00:00
Freddy 22062ba9da
Add default meta to test assertion () 2020-10-05 15:00:25 -06:00
Daniel Nephin 5a5fd4f0b1
Merge pull request from hashicorp/dnephin/extract-lib-retry
lib/retry - extract a new package from lib/retry.go
2020-10-05 14:22:37 -04:00
hashicorp-ci 3e9bd96c0a auto-updated agent/uiserver/bindata_assetfs.go from commit f26201a7a 2020-10-05 17:12:56 +00:00
freddygv 413a894a1a Do not evaluate discovery chain for topology upstreams 2020-10-05 10:24:50 -06:00
freddygv cf7b7fcdd6 Single DB txn for ServiceTopology and other PR comments 2020-10-05 10:24:50 -06:00
freddygv 7c26a71b4b Add topology HTTP endpoint 2020-10-05 10:24:50 -06:00
freddygv dbbf6b2e46 Add topology RPC endpoint 2020-10-05 10:24:50 -06:00
freddygv 98c81976f5 Add topology ACL filter 2020-10-05 10:24:50 -06:00
freddygv f906b94351 Add func to combine up+downstream queries 2020-10-05 10:24:50 -06:00
freddygv 5c913ec312 factor in discovery chain when querying up/downstreams 2020-10-05 10:24:50 -06:00
freddygv b012d8374e support querying upstreams/downstreams from registrations 2020-10-05 10:24:50 -06:00
freddygv a86cf88a4a Add method for downstreams from disco chain 2020-10-05 10:24:50 -06:00
hashicorp-ci 896d143647 auto-updated agent/uiserver/bindata_assetfs.go from commit 3a89c7d23 2020-10-05 14:07:13 +00:00
hashicorp-ci bf2b52f880 auto-updated agent/uiserver/bindata_assetfs.go from commit dfe57406d 2020-10-05 13:53:36 +00:00
hashicorp-ci 81baad4038 auto-updated agent/uiserver/bindata_assetfs.go from commit b871837ee 2020-10-05 13:14:40 +00:00
Daniel Nephin 529f252d5c rpcclient: Add health.Client and use it in http and dns
This new package provides a client agent implementation of an interface
for fetching the health of services.

This approach has a number of benefits:

1. It provides a much more explicit interface. Instead of everything
   dependency on `RPC()` and `Cache.Get()` for many unrelated things
   they can depend on a type that are named according to the behaviour
   it provides.

2. It gives us a single place to vary the behaviour and migrate to
   a new form of RPC (gRPC). The current implementation has two options
   (cache, or direct RPC), and in the future we will have more.
   It is also a great opporunity to start adding `context.Context` args
   to these operations, which in the future will allow us to cancel
   the operations.

3. As a concequence of the first, in the Server agent where we make
   these calls we can replace the current in-memory RPC calls with
   a thin adapter for the real method. This removes the `net/rpc`
   machinery from the call in places where it is not needed.

This new package is quite small right now, but I think we can expect it
to grow to a more reasonable size as other RPC calls are replaced.

This change also happens to replace two very similar implementations with
a single implementation.
2020-10-04 18:55:02 -04:00
Daniel Nephin e54567223b lib/retry: Refactor to reduce the interface surface
Reduce Jitter to one function

Rename NewRetryWaiter

Fix a bug in calculateWait where maxWait was applied before jitter, which would make it
possible to wait longer than maxWait.
2020-10-04 18:12:42 -04:00
Daniel Nephin ca26dfb4a2 lib/retry: extract a new package from lib 2020-10-04 17:43:01 -04:00
Kit Patella f5c51ae13b remove consul.api.http from filtered metric prefixes 2020-10-02 14:16:02 -07:00
Kit Patella 52451cf846
Merge pull request from coignetp/http-metrics-label
Use method and path as labels for http metrics
2020-10-02 13:41:48 -07:00
hashicorp-ci 90b5f1a838 auto-updated agent/uiserver/bindata_assetfs.go from commit 8b409529a 2020-10-02 19:28:38 +00:00
Daniel Nephin 1c6be5ac75 stream: full test coverage for EventPublisher.Subscribe 2020-10-02 13:46:24 -04:00
Daniel Nephin a5df5d17b4 stream: refactor to support change in framing events
Removing EndOfEmptySnapshot, add NewSnapshotToFollow
2020-10-02 13:41:31 -04:00
Daniel Nephin 04b51de783
Merge pull request from hashicorp/streaming/prep-for-subscribe-service
state: use protobuf Topic and and export payload type
2020-10-02 13:30:06 -04:00
Paul Banks d0c160130b
Merge pull request from hashicorp/ui-config-metrics
Add config changes for UI metrics
2020-10-01 17:38:03 +01:00
Paul Banks 3b8125a24d
Update all the references in CI and makefile to the bindata file location 2020-10-01 16:19:10 +01:00
R.B. Boyer 9a55c50694
ensure these tests work fine with namespaces in enterprise () 2020-10-01 09:54:46 -05:00
R.B. Boyer 9801ef8eb1
agent: enable enable_central_service_config by default () 2020-10-01 09:19:14 -05:00
Paul Banks 3ff5901be8
Fix ui dir where there is no index tests and lint issue. 2020-10-01 12:26:19 +01:00
Paul Banks e4db845246
Refactor uiserver to separate package, cleaner Reloading 2020-10-01 11:32:25 +01:00
R.B. Boyer 237a7a0da0
server: ensure that we also shutdown network segment serf instances on server shutdown ()
This really only matters for unit tests, since typically if an agent shuts down its server, it follows that up by exiting the process, which would also clean up all of the networking anyway.
2020-09-30 16:23:43 -05:00
Kyle Havlovitz 2ec94b027e connect: Enable renewing the intermediate cert in the primary DC 2020-09-30 12:31:21 -07:00
Paul Banks f6d55e1d25
Fix reload test; address other PR feedback 2020-09-30 18:00:07 +01:00
Paul Banks 54a33efa4b
Fix JSON encoding of metrics options which broke the index but didn't break tests.
Also add tests that do catch that error.
2020-09-30 17:59:19 +01:00
Paul Banks 526bab6164
Add config changes for UI metrics 2020-09-30 17:59:16 +01:00
hashicorp-ci b6406df147 auto-updated agent/bindata_assetfs.go from commit 1a6f3d524 2020-09-30 15:28:06 +00:00
hashicorp-ci 1a6f3d5246 auto-updated agent/bindata_assetfs.go from commit 8e174cae6 2020-09-30 15:23:27 +00:00
hashicorp-ci 8e174cae66 auto-updated agent/bindata_assetfs.go from commit 823d6dadb 2020-09-30 15:17:41 +00:00
Aliaksandr Mianzhynski 74cfba7065 Fix GRPCUseTLS flag HTTP API mapping 2020-09-29 18:29:56 +03:00
freddygv 9fa1b13df9 Resolve conflicts 2020-09-29 08:59:18 -06:00
Daniel Nephin b7ca15e910 stream: move goroutine out of New
This change will make it easier to manage goroutine lifecycle from the caller.

Also expose EventPublisher from state.Store
2020-09-28 18:40:10 -04:00
Daniel Nephin 0fb2a5b992 state: use pbsubscribe.Topic for topic values 2020-09-28 18:40:10 -04:00
Daniel Nephin 7b1534ef05 state: rename and export EventPayload
The subscribe endpoint needs to be able to inspect the payload to filter
events, and convert them into the protobuf types.

Use the protobuf CatalogOp type for the operation field, for now. In the
future if we end up with multiple interfaces we should be able to remove
the protobuf dependency by changing this to an int32 and adding a test
for the mapping between the values.

Make the value of the payload a concrete type instead of interface{}. We
can create other payloads for other event types.
2020-09-28 18:34:30 -04:00
Daniel Nephin 6200325e3b
Merge pull request from amenzhinsky/grpc-hc-error
Return grpc serving status in health check errors
2020-09-25 13:24:32 -04:00
Hans Hasselberg 98d7ea82bd
fix ent error () 2020-09-25 10:31:42 -05:00
R.B. Boyer 7eef25daf5
agent: when enable_central_service_config is enabled ensure agent reload doesn't revert check state to critical ()
Likely introduced when  landed.
2020-09-24 16:24:04 -05:00
R.B. Boyer 0064f1936e
server: make sure that the various replication loggers use consistent logging () 2020-09-24 15:49:38 -05:00
R.B. Boyer 0fb088aac3
agent: make the json/hcl decoding of ConnectProxyConfig fully work with CamelCase and snake_case ()
Fixes 
2020-09-24 13:58:52 -05:00
Daniel Nephin f14145e6d9 agent/grpc: always close the conn when dialing fails. 2020-09-24 12:53:14 -04:00
Daniel Nephin e6ffd987a3 agent/grpc: seed the rand for shuffling servers 2020-09-24 12:53:14 -04:00
Daniel Nephin 2294793357 agent/grpc: use router.Manager to handle the rebalance
The router.Manager is already rebalancing servers for other connection pools, so it can call into our resolver to do the same.
This change allows us to remove the serf dependency from resolverBuilder, and remove Datacenter from the config.

Also revert the change to refreshServerRebalanceTimer
2020-09-24 12:53:14 -04:00
Daniel Nephin 2273673500 grpc: restore integration tests for grpc client conn pool
Add a fake rpc Listener
2020-09-24 12:53:14 -04:00
Daniel Nephin 07b4507f1e router: remove grpcServerTracker from managers
It only needs to be refereced from the Router, because there is only 1 instance, and the
Router can call AddServer/RemoveServer like it does on the Manager.
2020-09-24 12:53:14 -04:00
Daniel Nephin bad4d3ff7c grpc: redeuce dependencies, unexport, and add godoc
Rename GRPCClient to ClientConnPool. This type appears to be more of a
conn pool than a client. The clients receive the connections from this
pool.

Reduce some dependencies by adjusting the interface baoundaries.

Remove the need to create a second slice of Servers, just to pick one and throw the rest away.

Unexport serverResolver, it is not used outside the package.

Use a RWMutex for ServerResolverBuilder, some locking is read-only.

Add more godoc.
2020-09-24 12:53:10 -04:00
Daniel Nephin 25f47b46e1 grpc: move client conn pool to grpc package 2020-09-24 12:48:12 -04:00
Daniel Nephin f936ca5aea grpc: client conn pool and resolver
Extracted from 936522a13c

Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-24 12:46:22 -04:00
Daniel Nephin c18516ad7d
Merge pull request from hashicorp/dnephin/replace-consul-opts-with-base-deps
agent: Repalce ConsulOptions with a new struct from agent.BaseDeps
2020-09-24 12:45:54 -04:00
Paul Banks 7d58901ae8
Fix bad int -> string conversions caught by go vet changes in 1.15 () 2020-09-24 11:14:07 +01:00
Alexander Mykolaichuk af753ee6a5
added permission denied error message () 2020-09-22 20:36:07 +02:00
Hans Hasselberg a89ee1a7ca
use service datacenter for dns name ()
* Use args.Datacenter instead of configured datacenter
2020-09-22 20:34:09 +02:00
Aliaksandr Mianzhynski 2c6fd6b796 Return grpc serving status in health check errors 2020-09-22 21:16:58 +03:00
Daniel Nephin 282fbdfa75 api: rename HTTPServer to HTTPHandlers
Resolves a TODO about naming. This type is a set of handlers for an http.Server, it is not
itself a Server. It provides http.Handler functions.
2020-09-18 17:38:23 -04:00
Hans Hasselberg d4877f03e7
fix TestLeader_SecondaryCA_IntermediateRenew ()
* fix lessThanHalfTime
* get lock for CAProvider()
* make a var to relate both vars
* rename to getCAProviderWithLock
* move CertificateTimeDriftBuffer to agent/connect/ca
2020-09-18 10:13:29 +02:00
Daniel Nephin ed6a0ebe4d
Merge pull request from hashicorp/dnephin/better-impl-of-TestAgent.HTTPAddr
http: fix tests incorrectly using HTTPAddr to get the address of the https server
2020-09-17 11:48:57 -04:00
Mike Morris 6b62751921
test: update tags for database service registrations and queries () 2020-09-16 14:05:01 -04:00
Kyle Havlovitz 1d22a0bc51
Merge pull request from hashicorp/vault-ca-renew-token
Automatically renew the token used by the Vault CA provider
2020-09-16 07:30:30 -07:00
Daniel Nephin 3995cc3408
Merge pull request from pierresouchay/do_not_flood_logs_with_Non-server_in_server-only_area
[BUGFIX] Avoid GetDatacenter* methods to flood Consul servers logs
2020-09-15 17:57:05 -04:00
Kyle Havlovitz b1b21139ca Merge branch 'master' into vault-ca-renew-token 2020-09-15 14:39:04 -07:00
Daniel Nephin cdd392d77f agent/consul: pass dependencies directly from agent
In an upcoming change we will need to pass a grpc.ClientConnPool from
BaseDeps into Server. While looking at that change I noticed all of the
existing consulOption fields are already on BaseDeps.

Instead of duplicating the fields, we can create a struct used by
agent/consul, and use that struct in BaseDeps. This allows us to pass
along dependencies without translating them into different
representations.

I also looked at moving all of BaseDeps in agent/consul, however that
created some circular imports. Resolving those cycles wouldn't be too
bad (it was only an error in agent/consul being imported from
cache-types), however this change seems a little better by starting to
introduce some structure to BaseDeps.

This change is also a small step in reducing the scope of Agent.

Also remove some constants that were only used by tests, and move the
relevant comment to where the live configuration is set.

Removed some validation from NewServer and NewClient, as these are not
really runtime errors. They would be code errors, which will cause a
panic anyway, so no reason to handle them specially here.
2020-09-15 17:29:32 -04:00
Daniel Nephin 3aa9bd4c23 agent/consul: make router required 2020-09-15 17:26:26 -04:00
Daniel Nephin d5edce269e
Merge pull request from hashicorp/streaming/fix-TestHandler_EmitsStats
streaming: Fix TestHandler_EmitsStats
2020-09-15 17:04:55 -04:00
Kyle Havlovitz 1cd7c43544 Update vault CA for latest api client 2020-09-15 13:33:55 -07:00
Paul Banks 2ae5230851
Update UI Config passing to not use an inline script ()
* Update UI Config passing to not use an inline script

* Update agent/http.go

* Fix incorrect placeholder name
2020-09-15 20:57:37 +01:00
Kyle Havlovitz 7ffef62ed7 Clean up CA shutdown logic and error 2020-09-15 12:28:58 -07:00
Kyle Havlovitz 35bb09f85c
Merge pull request from hashicorp/common-intermediate-ttl
Move IntermediateCertTTL to common CA config
2020-09-15 12:03:29 -07:00
Pierre Souchay 638dcd3360 [BUGFIX] Avoid GetDatacenter* methods to flood Consul servers logs
When calling `GetDatacentersByDistance()` or `GetDatacentersMap()`, an
incorrect condition was used to diplay log message, thus flooding
Consul's logs.

Example of message:

```
  [WARN] agent.router: Non-server in server-only area: non_server=myClientNode area=lan
```

This message is only valid for WAN areas, filter to avoid creating
hundreds of logs/s on our clusters, each time someone is calling this
method.

Our logs were flooded by such messages when migrating our Consul servers
from 1.7.7 to 1.8.4.

This will issue fix 
2020-09-15 11:54:59 +02:00
Daniel Nephin 636f76f6f1 agent/grpc: make TestHandler_EmitsStats predictable
Occasionally this test would flake. The flakes were fixed by:

1. Stopping the service and retrying to check on metrics. This way we
   also include the active_streams going to 0 in the metric calls.

2. Using a reference to the global Metrics. This way when other tests
   have background goroutines that are still shutting down, they won't
   emit metrics to the metric instance with the fake Sink. The stats
   test can patch the local reference to the global, so the existing
   statHandlers will continue to emit to the global, but the stats
   test will send all metrics to the replacement.
2020-09-14 19:05:22 -04:00
Daniel Nephin ee65ee541e grpc: add Datacenter field to testing service response 2020-09-14 19:02:09 -04:00
freddygv 856d5a25ee Fix text type assertion 2020-09-14 16:28:40 -06:00
freddygv 7fd518ff1d Merge master 2020-09-14 16:17:43 -06:00
freddygv 87541ab80a Fix type assertion 2020-09-14 16:12:21 -06:00
Daniel Nephin 20aea3dbc9
Merge pull request from hashicorp/streaming/add-grpc-server
streaming: add gRPC server for handling connections
2020-09-14 15:24:54 -04:00
freddygv 7b9d1b41d5 Resolve conflicts against master 2020-09-11 18:41:58 -06:00
freddygv 768dbaa68d Add session flag to cookie config 2020-09-11 18:34:03 -06:00
freddygv 9d2a9169fd PR comments 2020-09-11 10:49:26 -06:00
Kyle Havlovitz 49056fe70f Clean up Vault renew tests and shutdown 2020-09-11 08:41:05 -07:00
freddygv eab90ea9fa Revert EnvoyConfig nesting 2020-09-11 09:21:43 -06:00
Kyle Havlovitz f40fb577fe Use mapstructure for decoding vault data 2020-09-10 06:31:04 -07:00
Kyle Havlovitz aa97366020 Add a stop function to make sure the renewer is shut down on leader change 2020-09-10 06:12:48 -07:00
Kyle Havlovitz 2f7210bde2 Move IntermediateCertTTL to common CA config 2020-09-10 00:23:22 -07:00
Kyle Havlovitz 411b6537ef Add a test for token renewal 2020-09-09 16:36:37 -07:00
Daniel Nephin fd42804063 grpc: Add a simple test service for testing the gRPC server 2020-09-08 12:10:43 -04:00
Daniel Nephin 2257247095 server: add gRPC server for streaming events
Includes a stats handler and stream interceptor for grpc metrics.

Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-08 12:10:41 -04:00
Daniel Nephin 0bb9c318b7 http: fix tests incorrectly using HTTPAddr to get the address of the
https server.

In  I changed a few tests to use TestAgent.HTTPAddr() to find the
addr used in the test. Due to the way HTTPAddr() was implemented these
tests were passing, but I think the pass was incidental. HTTPAddr() was
not matching any servers, and was instead returning the last server,
which happened to be the one these tests wanted.

This commit fixes the implementation of HTTPAddr to panic if no match
was found. The tests which require an HTTPS server are changed to use
a new firstAddr() to look up the correct address.
2020-09-04 15:29:17 -04:00
freddygv 403a180430 Set tgw filter router config name to cluster name 2020-09-04 12:45:05 -06:00
Hans Hasselberg 436a7032d1
secondaryIntermediateCertRenewalWatch abort on success ()
secondaryIntermediateCertRenewalWatch was using `retryLoopBackoff` to
renew the intermediate certificate. Once it entered the inner loop and
started `retryLoopBackoff` it would never leave that.
`retryLoopBackoffAbortOnSuccess` will return when renewing is
successful, like it was intended originally.
2020-09-04 11:47:16 +02:00
freddygv 959d9913b8 Add server receiver to routes and log tgw err 2020-09-03 16:19:58 -06:00
Daniel Nephin ed4b51f1ae
Merge pull request from hashicorp/streaming/add-service-health-events
streaming: add ServiceHealth events
2020-09-03 17:53:56 -04:00
Daniel Nephin 4c9ed41eab
Merge pull request from hashicorp/dnephin/agent-setup-persisted-tokens
agent: move token persistence from agent into token.Store
2020-09-03 17:29:21 -04:00
Daniel Nephin e573e64d58 state: handle terminating gateways in service health events 2020-09-03 16:58:05 -04:00
Daniel Nephin 3775392fb5 state: improve comments in catalog_events.go
Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-03 16:58:05 -04:00
Daniel Nephin 417c5c93a8 state: use changeType in serviceChanges
To be a little more explicit, instead of nil implying an indirect change
2020-09-03 16:58:05 -04:00
Daniel Nephin 01424ba146 don't over allocate slice 2020-09-03 16:58:04 -04:00
Daniel Nephin d210242875 state: fix a bug in building service health events
The nodeCheck slice was being used as the first arg in append, which in some cases will modify the array backing the slice. This would lead to service checks for other services in the wrong event.

Also refactor some things to reduce the arguments to functions.
2020-09-03 16:58:04 -04:00
Daniel Nephin 7581305523 state: Remove unused args and return values
Also rename some functions to identify them as constructors for events
2020-09-03 16:58:04 -04:00
Daniel Nephin 27b02d391c state: use an enum for tracking node changes 2020-09-03 16:58:04 -04:00
Daniel Nephin 09329b542d state: serviceHealthSnapshot
refactored to remove unused return value and remove duplication
2020-09-03 16:58:04 -04:00