Commit Graph

2654 Commits

Author SHA1 Message Date
Paul Banks f11b759ddf
Actually proxy the query string too 2020-10-09 12:27:24 +01:00
Daniel Nephin 3ff6c5b3d3 cache-types: skip tests with races 2020-10-08 20:15:13 -04:00
Daniel Nephin af8a617797 grpc: fix data rate in stats handler test 2020-10-08 19:43:49 -04:00
Freddy 13df5d5bf8
Add protocol to the topology endpoint response (#8868) 2020-10-08 17:31:54 -06:00
Daniel Nephin c5d57c9f07 subscribe: add test cases for newEventFromStreamEvent 2020-10-08 18:48:17 -04:00
Kit Patella adeabf2399
Merge pull request #8877 from hashicorp/mkcp/telemetry/consul.api.http
Add flag for disabling 1.9 metrics backwards compatibility and warnings when set to default
2020-10-08 13:22:37 -07:00
R.B. Boyer 46a4ef7248
agent: allow the /v1/connect/intentions/match endpoint to use the agent cache (#8875)
This is the recommended proxy integration API for listing intentions
which should not require an active connection to the servers to resolve
after the initial cache filling.
2020-10-08 14:51:53 -05:00
Daniel Nephin f185124320 subscribe: Add steps to rpc/subscribe tests
To make them easier to follow
2020-10-08 15:38:01 -04:00
hashicorp-ci de99c78f14 auto-updated agent/uiserver/bindata_assetfs.go from commit 13dfde75a 2020-10-08 19:36:48 +00:00
Matt Keeler 38f5ddce2a
Add per-agent reconnect timeouts (#8781)
This allows for client agent to be run in a more stateless manner where they may be abruptly terminated and not expected to come back. If advertising a per-agent reconnect timeout using the advertise_reconnect_timeout configuration when that agent leaves, other agents will wait only that amount of time for the agent to come back before reaping it.

This has the advantageous side effect of causing servers to deregister the node/services/checks for that agent sooner than if the global reconnect_timeout was used.
2020-10-08 15:02:19 -04:00
Paul Banks 332477f24c
Add /v1/internal/ui/metrics-proxy API endpoint that proxies to a configured metrics provider backend. 2020-10-08 17:32:29 +01:00
Paul Banks 80910b46f3
Fix merge conflicts 2020-10-08 17:31:36 +01:00
Daniel Nephin b93577c94f config: add field for enabling streaming RPC endpoint 2020-10-08 12:11:20 -04:00
Daniel Nephin b5b790d4c0 streaming: Use a shorter LastGetTTL for the cache 2020-10-08 12:11:20 -04:00
hashicorp-ci fc6a7df5f6 auto-updated agent/uiserver/bindata_assetfs.go from commit b373456c7 2020-10-08 15:56:05 +00:00
Freddy 164ce57db2
Support ingress gateways in mesh viz endpoint (#8864)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-08 09:47:09 -06:00
hashicorp-ci 418fe6ba32 auto-updated agent/uiserver/bindata_assetfs.go from commit ef7b1f8a4 2020-10-08 15:41:40 +00:00
hashicorp-ci 12e735dbca auto-updated agent/uiserver/bindata_assetfs.go from commit d849f025c 2020-10-08 15:08:16 +00:00
Daniel Nephin fcaf362c9f
Merge pull request #8841 from hashicorp/streaming/materialize-view-fix-service-unique-id
streaming: Use an ID that includes namespace to store services in the materialized view
2020-10-07 21:28:53 -04:00
Daniel Nephin 2513f42c68
Merge pull request #8809 from hashicorp/streaming/materialize-view
Add StreamingHealthServices cache-type
2020-10-07 21:26:38 -04:00
Daniel Nephin b103568e98
Merge pull request #8818 from hashicorp/streaming/add-subscribe-service-batch-events
stream: handle batch events as a special case of Event
2020-10-07 21:25:32 -04:00
Daniel Nephin da6400192b
Merge pull request #8768 from hashicorp/streaming/add-subscribe-service
subscribe: add subscribe service for streaming change events
2020-10-07 21:24:03 -04:00
Freddy da91e999f6
Return intention info in svc topology endpoint (#8853) 2020-10-07 18:35:34 -06:00
Kit Patella 7fe2f80b4b add config flag to disable 1.9 metrics backwards compatibility. Add warnings on start and reload on default value 2020-10-07 17:12:52 -07:00
Daniel Nephin 21c21191f4 structs: add CheckServiceNode.CanRead
And use it from the subscribe endpoint.
2020-10-07 18:15:13 -04:00
hashicorp-ci 8655b24639 auto-updated agent/uiserver/bindata_assetfs.go from commit 0d0de4309 2020-10-07 14:13:29 +00:00
hashicorp-ci eb4bc23a03 auto-updated agent/uiserver/bindata_assetfs.go from commit a61b6c53d 2020-10-07 13:50:39 +00:00
hashicorp-ci aa0f5ff839 auto-updated agent/uiserver/bindata_assetfs.go from commit 222555c4c 2020-10-07 08:09:54 +00:00
R.B. Boyer 1b413b0444
connect: support defining intentions using layer 7 criteria (#8839)
Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).
2020-10-06 17:09:13 -05:00
Daniel Nephin 8b887af0d3 streaming: store services with a unique ID that includes namespace 2020-10-06 16:54:56 -04:00
R.B. Boyer a2a8e9c783
connect: intentions are now managed as a new config entry kind "service-intentions" (#8834)
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
Daniel Nephin 5972bdc87c streaming: improve godoc for cache-type
And fix a bug where any error that implemented the temporary interface was considered
a temporary error, even when the method would return false.
2020-10-06 13:52:02 -04:00
Daniel Nephin d0b87cd5d0 submatview: remove notifyUpdateLocked from reset
This call appears to only be necessary because reset() was called from
NewMaterializer.

This commit has the constructor set a default value for updateCh, and
removes both the call to reset() from New(), and the call to
notifyUpdateLocked() from reset().

This should ensure that we do not notify the Fetch() call before we have new
values to report.
2020-10-06 13:22:02 -04:00
Daniel Nephin 5eab1d8cef submatview: remove method receiver from handlers 2020-10-06 13:22:02 -04:00
Daniel Nephin 3fa08beecf submatview: add a test for handling of NewSnapshotToFollow
Also add some godoc
Rename some vars and functions
Fix a data race in the new cache test for entry closing.
2020-10-06 13:22:02 -04:00
Daniel Nephin 534d8b45bb submatview: refactor Materializer
Refactor of Materializer.Run
Use handlers to manage state in Materializer
Rename Materializer receiver
rename m.l to m.lock, and flip some conditionals to remove the negative.
Improve godoc, rename Deps, move resetErr, and pass err into notifyUpdate
Update for NewSnapshotToFollow events
Refactor to move context cancel out of Materializer
2020-10-06 13:22:02 -04:00
Daniel Nephin e849f6d7ac submatview: Move the 'use materialize from result.State' logic
No need to do all this other work if we have one already.

This logic moved closer to this call site 3 times during the process
of refactoring.
2020-10-06 13:22:02 -04:00
Daniel Nephin edf30b2714 submatview: Move Materializer to submatview package 2020-10-06 13:22:02 -04:00
Daniel Nephin ed45957ffb submatview: Refactor MaterializeView
Replace InitFilter with Reset.
Removes the need to store a fatalErr and the cache-type, and removes the need to recreate the filter
each time.
Pass dependencies into MaterializedView.
Remove context from MaterializedView.
Rename state to view.
Rename MaterialziedView to Materialzier.
Rename to NewMaterializer
Pass in retry.Waiter
2020-10-06 13:22:02 -04:00
Daniel Nephin b576a2d3c7 cache-types: Update Streaming health cache-type
To use latest protobuf types
2020-10-06 13:22:02 -04:00
Daniel Nephin 132b76acef agent/cache: Add cache-type and materialized view for streaming health
Extracted from d97412ce4c

Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 13:21:57 -04:00
Daniel Nephin b27068b72a stream: Return a single event from a subscription.Next
Handle batch events as a single event
2020-10-06 13:18:20 -04:00
Daniel Nephin f5d11562f2 subscribe: update to use NewSnapshotToFollow event 2020-10-06 12:49:35 -04:00
Daniel Nephin e3290f5971 Move agent/subscribe -> agent/rpc/subscribe 2020-10-06 12:49:35 -04:00
Daniel Nephin dbb8bd679f subscirbe: extract streamID and logging from Subscribe
By extracting all of the tracing logic the core logic of the Subscribe
endpoint is much easier to read.
2020-10-06 12:49:35 -04:00
Daniel Nephin 9e4ebacb05 subscribe: add integration test for acl token updates 2020-10-06 12:49:35 -04:00
Daniel Nephin 39beed0af6 subscribe: add integration test for filtering events by acl 2020-10-06 12:49:35 -04:00
Daniel Nephin 083f4e8f57 subscribe: Add an integration test for forward to DC 2020-10-06 12:49:35 -04:00
Daniel Nephin 013ababda4 subscribe: add first integration test for Server 2020-10-06 12:49:35 -04:00
Daniel Nephin d0256a0c07 subscribe: add a stateless subscribe service for the gRPC server
With a Backend that provides access to the necessary dependencies.
2020-10-06 12:49:35 -04:00
Daniel Nephin f4ea3066fb subscribe: add commented out test cases
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin 106d781dc9 subscribe: add initial impl from streaming-rpc-final branch
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin 364f6589c8
Merge pull request #8799 from hashicorp/streaming/rename-framing-events
stream: remove EndOfEmptySnapshot, add NewSnapshotToFollow
2020-10-06 12:42:58 -04:00
Daniel Nephin a5c50c982d
Merge pull request #8806 from hashicorp/dnephin/service-health-interface
rpcclient: Add health.Client and use it in http and dns
2020-10-06 12:13:09 -04:00
R.B. Boyer 4998a08c56
server: create new memdb table for storing system metadata (#8703)
This adds a new very tiny memdb table and corresponding raft operation
for updating a very small effective map[string]string collection of
"system metadata". This can persistently record a fact about the Consul
state machine itself.

The first use of this feature will come in a later PR.
2020-10-06 10:08:37 -05:00
hashicorp-ci 356998d91e auto-updated agent/uiserver/bindata_assetfs.go from commit a7038b48c 2020-10-06 13:43:27 +00:00
hashicorp-ci bc5ca8b638 auto-updated agent/uiserver/bindata_assetfs.go from commit 5a39be47d 2020-10-06 13:32:27 +00:00
hashicorp-ci 36d219e2cc auto-updated agent/uiserver/bindata_assetfs.go from commit 76d3909f3 2020-10-06 08:42:39 +00:00
hashicorp-ci 4c9630ffad auto-updated agent/uiserver/bindata_assetfs.go from commit e4a0dcf10 2020-10-06 08:36:18 +00:00
Freddy 22062ba9da
Add default meta to test assertion (#8824) 2020-10-05 15:00:25 -06:00
Daniel Nephin 5a5fd4f0b1
Merge pull request #8802 from hashicorp/dnephin/extract-lib-retry
lib/retry - extract a new package from lib/retry.go
2020-10-05 14:22:37 -04:00
hashicorp-ci 3e9bd96c0a auto-updated agent/uiserver/bindata_assetfs.go from commit f26201a7a 2020-10-05 17:12:56 +00:00
freddygv 413a894a1a Do not evaluate discovery chain for topology upstreams 2020-10-05 10:24:50 -06:00
freddygv cf7b7fcdd6 Single DB txn for ServiceTopology and other PR comments 2020-10-05 10:24:50 -06:00
freddygv 7c26a71b4b Add topology HTTP endpoint 2020-10-05 10:24:50 -06:00
freddygv dbbf6b2e46 Add topology RPC endpoint 2020-10-05 10:24:50 -06:00
freddygv 98c81976f5 Add topology ACL filter 2020-10-05 10:24:50 -06:00
freddygv f906b94351 Add func to combine up+downstream queries 2020-10-05 10:24:50 -06:00
freddygv 5c913ec312 factor in discovery chain when querying up/downstreams 2020-10-05 10:24:50 -06:00
freddygv b012d8374e support querying upstreams/downstreams from registrations 2020-10-05 10:24:50 -06:00
freddygv a86cf88a4a Add method for downstreams from disco chain 2020-10-05 10:24:50 -06:00
hashicorp-ci 896d143647 auto-updated agent/uiserver/bindata_assetfs.go from commit 3a89c7d23 2020-10-05 14:07:13 +00:00
hashicorp-ci bf2b52f880 auto-updated agent/uiserver/bindata_assetfs.go from commit dfe57406d 2020-10-05 13:53:36 +00:00
hashicorp-ci 81baad4038 auto-updated agent/uiserver/bindata_assetfs.go from commit b871837ee 2020-10-05 13:14:40 +00:00
Daniel Nephin 529f252d5c rpcclient: Add health.Client and use it in http and dns
This new package provides a client agent implementation of an interface
for fetching the health of services.

This approach has a number of benefits:

1. It provides a much more explicit interface. Instead of everything
   dependency on `RPC()` and `Cache.Get()` for many unrelated things
   they can depend on a type that are named according to the behaviour
   it provides.

2. It gives us a single place to vary the behaviour and migrate to
   a new form of RPC (gRPC). The current implementation has two options
   (cache, or direct RPC), and in the future we will have more.
   It is also a great opporunity to start adding `context.Context` args
   to these operations, which in the future will allow us to cancel
   the operations.

3. As a concequence of the first, in the Server agent where we make
   these calls we can replace the current in-memory RPC calls with
   a thin adapter for the real method. This removes the `net/rpc`
   machinery from the call in places where it is not needed.

This new package is quite small right now, but I think we can expect it
to grow to a more reasonable size as other RPC calls are replaced.

This change also happens to replace two very similar implementations with
a single implementation.
2020-10-04 18:55:02 -04:00
Daniel Nephin e54567223b lib/retry: Refactor to reduce the interface surface
Reduce Jitter to one function

Rename NewRetryWaiter

Fix a bug in calculateWait where maxWait was applied before jitter, which would make it
possible to wait longer than maxWait.
2020-10-04 18:12:42 -04:00
Daniel Nephin ca26dfb4a2 lib/retry: extract a new package from lib 2020-10-04 17:43:01 -04:00
Kit Patella f5c51ae13b remove consul.api.http from filtered metric prefixes 2020-10-02 14:16:02 -07:00
Kit Patella 52451cf846
Merge pull request #8271 from coignetp/http-metrics-label
Use method and path as labels for http metrics
2020-10-02 13:41:48 -07:00
hashicorp-ci 90b5f1a838 auto-updated agent/uiserver/bindata_assetfs.go from commit 8b409529a 2020-10-02 19:28:38 +00:00
Daniel Nephin 1c6be5ac75 stream: full test coverage for EventPublisher.Subscribe 2020-10-02 13:46:24 -04:00
Daniel Nephin a5df5d17b4 stream: refactor to support change in framing events
Removing EndOfEmptySnapshot, add NewSnapshotToFollow
2020-10-02 13:41:31 -04:00
Daniel Nephin 04b51de783
Merge pull request #8769 from hashicorp/streaming/prep-for-subscribe-service
state: use protobuf Topic and and export payload type
2020-10-02 13:30:06 -04:00
Paul Banks d0c160130b
Merge pull request #8694 from hashicorp/ui-config-metrics
Add config changes for UI metrics
2020-10-01 17:38:03 +01:00
Paul Banks 3b8125a24d
Update all the references in CI and makefile to the bindata file location 2020-10-01 16:19:10 +01:00
R.B. Boyer 9a55c50694
ensure these tests work fine with namespaces in enterprise (#8794) 2020-10-01 09:54:46 -05:00
R.B. Boyer 9801ef8eb1
agent: enable enable_central_service_config by default (#8746) 2020-10-01 09:19:14 -05:00
Paul Banks 3ff5901be8
Fix ui dir where there is no index tests and lint issue. 2020-10-01 12:26:19 +01:00
Paul Banks e4db845246
Refactor uiserver to separate package, cleaner Reloading 2020-10-01 11:32:25 +01:00
R.B. Boyer 237a7a0da0
server: ensure that we also shutdown network segment serf instances on server shutdown (#8786)
This really only matters for unit tests, since typically if an agent shuts down its server, it follows that up by exiting the process, which would also clean up all of the networking anyway.
2020-09-30 16:23:43 -05:00
Kyle Havlovitz 2ec94b027e connect: Enable renewing the intermediate cert in the primary DC 2020-09-30 12:31:21 -07:00
Paul Banks f6d55e1d25
Fix reload test; address other PR feedback 2020-09-30 18:00:07 +01:00
Paul Banks 54a33efa4b
Fix JSON encoding of metrics options which broke the index but didn't break tests.
Also add tests that do catch that error.
2020-09-30 17:59:19 +01:00
Paul Banks 526bab6164
Add config changes for UI metrics 2020-09-30 17:59:16 +01:00
hashicorp-ci b6406df147 auto-updated agent/bindata_assetfs.go from commit 1a6f3d524 2020-09-30 15:28:06 +00:00
hashicorp-ci 1a6f3d5246 auto-updated agent/bindata_assetfs.go from commit 8e174cae6 2020-09-30 15:23:27 +00:00
hashicorp-ci 8e174cae66 auto-updated agent/bindata_assetfs.go from commit 823d6dadb 2020-09-30 15:17:41 +00:00
Aliaksandr Mianzhynski 74cfba7065 Fix GRPCUseTLS flag HTTP API mapping 2020-09-29 18:29:56 +03:00
freddygv 9fa1b13df9 Resolve conflicts 2020-09-29 08:59:18 -06:00
Daniel Nephin b7ca15e910 stream: move goroutine out of New
This change will make it easier to manage goroutine lifecycle from the caller.

Also expose EventPublisher from state.Store
2020-09-28 18:40:10 -04:00
Daniel Nephin 0fb2a5b992 state: use pbsubscribe.Topic for topic values 2020-09-28 18:40:10 -04:00
Daniel Nephin 7b1534ef05 state: rename and export EventPayload
The subscribe endpoint needs to be able to inspect the payload to filter
events, and convert them into the protobuf types.

Use the protobuf CatalogOp type for the operation field, for now. In the
future if we end up with multiple interfaces we should be able to remove
the protobuf dependency by changing this to an int32 and adding a test
for the mapping between the values.

Make the value of the payload a concrete type instead of interface{}. We
can create other payloads for other event types.
2020-09-28 18:34:30 -04:00
Daniel Nephin 6200325e3b
Merge pull request #8726 from amenzhinsky/grpc-hc-error
Return grpc serving status in health check errors
2020-09-25 13:24:32 -04:00
Hans Hasselberg 98d7ea82bd
fix ent error (#8750) 2020-09-25 10:31:42 -05:00
R.B. Boyer 7eef25daf5
agent: when enable_central_service_config is enabled ensure agent reload doesn't revert check state to critical (#8747)
Likely introduced when #7345 landed.
2020-09-24 16:24:04 -05:00
R.B. Boyer 0064f1936e
server: make sure that the various replication loggers use consistent logging (#8745) 2020-09-24 15:49:38 -05:00
R.B. Boyer 0fb088aac3
agent: make the json/hcl decoding of ConnectProxyConfig fully work with CamelCase and snake_case (#8741)
Fixes #7418
2020-09-24 13:58:52 -05:00
Daniel Nephin f14145e6d9 agent/grpc: always close the conn when dialing fails. 2020-09-24 12:53:14 -04:00
Daniel Nephin e6ffd987a3 agent/grpc: seed the rand for shuffling servers 2020-09-24 12:53:14 -04:00
Daniel Nephin 2294793357 agent/grpc: use router.Manager to handle the rebalance
The router.Manager is already rebalancing servers for other connection pools, so it can call into our resolver to do the same.
This change allows us to remove the serf dependency from resolverBuilder, and remove Datacenter from the config.

Also revert the change to refreshServerRebalanceTimer
2020-09-24 12:53:14 -04:00
Daniel Nephin 2273673500 grpc: restore integration tests for grpc client conn pool
Add a fake rpc Listener
2020-09-24 12:53:14 -04:00
Daniel Nephin 07b4507f1e router: remove grpcServerTracker from managers
It only needs to be refereced from the Router, because there is only 1 instance, and the
Router can call AddServer/RemoveServer like it does on the Manager.
2020-09-24 12:53:14 -04:00
Daniel Nephin bad4d3ff7c grpc: redeuce dependencies, unexport, and add godoc
Rename GRPCClient to ClientConnPool. This type appears to be more of a
conn pool than a client. The clients receive the connections from this
pool.

Reduce some dependencies by adjusting the interface baoundaries.

Remove the need to create a second slice of Servers, just to pick one and throw the rest away.

Unexport serverResolver, it is not used outside the package.

Use a RWMutex for ServerResolverBuilder, some locking is read-only.

Add more godoc.
2020-09-24 12:53:10 -04:00
Daniel Nephin 25f47b46e1 grpc: move client conn pool to grpc package 2020-09-24 12:48:12 -04:00
Daniel Nephin f936ca5aea grpc: client conn pool and resolver
Extracted from 936522a13c

Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-24 12:46:22 -04:00
Daniel Nephin c18516ad7d
Merge pull request #8680 from hashicorp/dnephin/replace-consul-opts-with-base-deps
agent: Repalce ConsulOptions with a new struct from agent.BaseDeps
2020-09-24 12:45:54 -04:00
Paul Banks 7d58901ae8
Fix bad int -> string conversions caught by go vet changes in 1.15 (#8739) 2020-09-24 11:14:07 +01:00
Alexander Mykolaichuk af753ee6a5
added permission denied error message (#8044) 2020-09-22 20:36:07 +02:00
Hans Hasselberg a89ee1a7ca
use service datacenter for dns name (#8704)
* Use args.Datacenter instead of configured datacenter
2020-09-22 20:34:09 +02:00
Aliaksandr Mianzhynski 2c6fd6b796 Return grpc serving status in health check errors 2020-09-22 21:16:58 +03:00
Daniel Nephin 282fbdfa75 api: rename HTTPServer to HTTPHandlers
Resolves a TODO about naming. This type is a set of handlers for an http.Server, it is not
itself a Server. It provides http.Handler functions.
2020-09-18 17:38:23 -04:00
Hans Hasselberg d4877f03e7
fix TestLeader_SecondaryCA_IntermediateRenew (#8702)
* fix lessThanHalfTime
* get lock for CAProvider()
* make a var to relate both vars
* rename to getCAProviderWithLock
* move CertificateTimeDriftBuffer to agent/connect/ca
2020-09-18 10:13:29 +02:00
Daniel Nephin ed6a0ebe4d
Merge pull request #8620 from hashicorp/dnephin/better-impl-of-TestAgent.HTTPAddr
http: fix tests incorrectly using HTTPAddr to get the address of the https server
2020-09-17 11:48:57 -04:00
Mike Morris 6b62751921
test: update tags for database service registrations and queries (#8693) 2020-09-16 14:05:01 -04:00
Kyle Havlovitz 1d22a0bc51
Merge pull request #8560 from hashicorp/vault-ca-renew-token
Automatically renew the token used by the Vault CA provider
2020-09-16 07:30:30 -07:00
Daniel Nephin 3995cc3408
Merge pull request #8685 from pierresouchay/do_not_flood_logs_with_Non-server_in_server-only_area
[BUGFIX] Avoid GetDatacenter* methods to flood Consul servers logs
2020-09-15 17:57:05 -04:00
Kyle Havlovitz b1b21139ca Merge branch 'master' into vault-ca-renew-token 2020-09-15 14:39:04 -07:00
Daniel Nephin cdd392d77f agent/consul: pass dependencies directly from agent
In an upcoming change we will need to pass a grpc.ClientConnPool from
BaseDeps into Server. While looking at that change I noticed all of the
existing consulOption fields are already on BaseDeps.

Instead of duplicating the fields, we can create a struct used by
agent/consul, and use that struct in BaseDeps. This allows us to pass
along dependencies without translating them into different
representations.

I also looked at moving all of BaseDeps in agent/consul, however that
created some circular imports. Resolving those cycles wouldn't be too
bad (it was only an error in agent/consul being imported from
cache-types), however this change seems a little better by starting to
introduce some structure to BaseDeps.

This change is also a small step in reducing the scope of Agent.

Also remove some constants that were only used by tests, and move the
relevant comment to where the live configuration is set.

Removed some validation from NewServer and NewClient, as these are not
really runtime errors. They would be code errors, which will cause a
panic anyway, so no reason to handle them specially here.
2020-09-15 17:29:32 -04:00
Daniel Nephin 3aa9bd4c23 agent/consul: make router required 2020-09-15 17:26:26 -04:00
Daniel Nephin d5edce269e
Merge pull request #8679 from hashicorp/streaming/fix-TestHandler_EmitsStats
streaming: Fix TestHandler_EmitsStats
2020-09-15 17:04:55 -04:00
Kyle Havlovitz 1cd7c43544 Update vault CA for latest api client 2020-09-15 13:33:55 -07:00
Paul Banks 2ae5230851
Update UI Config passing to not use an inline script (#8645)
* Update UI Config passing to not use an inline script

* Update agent/http.go

* Fix incorrect placeholder name
2020-09-15 20:57:37 +01:00
Kyle Havlovitz 7ffef62ed7 Clean up CA shutdown logic and error 2020-09-15 12:28:58 -07:00
Kyle Havlovitz 35bb09f85c
Merge pull request #8646 from hashicorp/common-intermediate-ttl
Move IntermediateCertTTL to common CA config
2020-09-15 12:03:29 -07:00
Pierre Souchay 638dcd3360 [BUGFIX] Avoid GetDatacenter* methods to flood Consul servers logs
When calling `GetDatacentersByDistance()` or `GetDatacentersMap()`, an
incorrect condition was used to diplay log message, thus flooding
Consul's logs.

Example of message:

```
  [WARN] agent.router: Non-server in server-only area: non_server=myClientNode area=lan
```

This message is only valid for WAN areas, filter to avoid creating
hundreds of logs/s on our clusters, each time someone is calling this
method.

Our logs were flooded by such messages when migrating our Consul servers
from 1.7.7 to 1.8.4.

This will issue fix #8663
2020-09-15 11:54:59 +02:00
Daniel Nephin 636f76f6f1 agent/grpc: make TestHandler_EmitsStats predictable
Occasionally this test would flake. The flakes were fixed by:

1. Stopping the service and retrying to check on metrics. This way we
   also include the active_streams going to 0 in the metric calls.

2. Using a reference to the global Metrics. This way when other tests
   have background goroutines that are still shutting down, they won't
   emit metrics to the metric instance with the fake Sink. The stats
   test can patch the local reference to the global, so the existing
   statHandlers will continue to emit to the global, but the stats
   test will send all metrics to the replacement.
2020-09-14 19:05:22 -04:00
Daniel Nephin ee65ee541e grpc: add Datacenter field to testing service response 2020-09-14 19:02:09 -04:00
freddygv 856d5a25ee Fix text type assertion 2020-09-14 16:28:40 -06:00
freddygv 7fd518ff1d Merge master 2020-09-14 16:17:43 -06:00
freddygv 87541ab80a Fix type assertion 2020-09-14 16:12:21 -06:00
Daniel Nephin 20aea3dbc9
Merge pull request #8587 from hashicorp/streaming/add-grpc-server
streaming: add gRPC server for handling connections
2020-09-14 15:24:54 -04:00
freddygv 7b9d1b41d5 Resolve conflicts against master 2020-09-11 18:41:58 -06:00
freddygv 768dbaa68d Add session flag to cookie config 2020-09-11 18:34:03 -06:00
freddygv 9d2a9169fd PR comments 2020-09-11 10:49:26 -06:00
Kyle Havlovitz 49056fe70f Clean up Vault renew tests and shutdown 2020-09-11 08:41:05 -07:00
freddygv eab90ea9fa Revert EnvoyConfig nesting 2020-09-11 09:21:43 -06:00
Kyle Havlovitz f40fb577fe Use mapstructure for decoding vault data 2020-09-10 06:31:04 -07:00
Kyle Havlovitz aa97366020 Add a stop function to make sure the renewer is shut down on leader change 2020-09-10 06:12:48 -07:00
Kyle Havlovitz 2f7210bde2 Move IntermediateCertTTL to common CA config 2020-09-10 00:23:22 -07:00
Kyle Havlovitz 411b6537ef Add a test for token renewal 2020-09-09 16:36:37 -07:00
Daniel Nephin fd42804063 grpc: Add a simple test service for testing the gRPC server 2020-09-08 12:10:43 -04:00
Daniel Nephin 2257247095 server: add gRPC server for streaming events
Includes a stats handler and stream interceptor for grpc metrics.

Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-08 12:10:41 -04:00
Daniel Nephin 0bb9c318b7 http: fix tests incorrectly using HTTPAddr to get the address of the
https server.

In #8234 I changed a few tests to use TestAgent.HTTPAddr() to find the
addr used in the test. Due to the way HTTPAddr() was implemented these
tests were passing, but I think the pass was incidental. HTTPAddr() was
not matching any servers, and was instead returning the last server,
which happened to be the one these tests wanted.

This commit fixes the implementation of HTTPAddr to panic if no match
was found. The tests which require an HTTPS server are changed to use
a new firstAddr() to look up the correct address.
2020-09-04 15:29:17 -04:00
freddygv 403a180430 Set tgw filter router config name to cluster name 2020-09-04 12:45:05 -06:00
Hans Hasselberg 436a7032d1
secondaryIntermediateCertRenewalWatch abort on success (#8588)
secondaryIntermediateCertRenewalWatch was using `retryLoopBackoff` to
renew the intermediate certificate. Once it entered the inner loop and
started `retryLoopBackoff` it would never leave that.
`retryLoopBackoffAbortOnSuccess` will return when renewing is
successful, like it was intended originally.
2020-09-04 11:47:16 +02:00
freddygv 959d9913b8 Add server receiver to routes and log tgw err 2020-09-03 16:19:58 -06:00
Daniel Nephin ed4b51f1ae
Merge pull request #8357 from hashicorp/streaming/add-service-health-events
streaming: add ServiceHealth events
2020-09-03 17:53:56 -04:00
Daniel Nephin 4c9ed41eab
Merge pull request #8554 from hashicorp/dnephin/agent-setup-persisted-tokens
agent: move token persistence from agent into token.Store
2020-09-03 17:29:21 -04:00
Daniel Nephin e573e64d58 state: handle terminating gateways in service health events 2020-09-03 16:58:05 -04:00
Daniel Nephin 3775392fb5 state: improve comments in catalog_events.go
Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-03 16:58:05 -04:00
Daniel Nephin 417c5c93a8 state: use changeType in serviceChanges
To be a little more explicit, instead of nil implying an indirect change
2020-09-03 16:58:05 -04:00
Daniel Nephin 01424ba146 don't over allocate slice 2020-09-03 16:58:04 -04:00
Daniel Nephin d210242875 state: fix a bug in building service health events
The nodeCheck slice was being used as the first arg in append, which in some cases will modify the array backing the slice. This would lead to service checks for other services in the wrong event.

Also refactor some things to reduce the arguments to functions.
2020-09-03 16:58:04 -04:00
Daniel Nephin 7581305523 state: Remove unused args and return values
Also rename some functions to identify them as constructors for events
2020-09-03 16:58:04 -04:00
Daniel Nephin 27b02d391c state: use an enum for tracking node changes 2020-09-03 16:58:04 -04:00
Daniel Nephin 09329b542d state: serviceHealthSnapshot
refactored to remove unused return value and remove duplication
2020-09-03 16:58:04 -04:00
Daniel Nephin bf523420ee state: Add Change processor and snapshotter for service health
Co-authored-by: Paul Banks <banks@banksco.de>
2020-09-03 16:58:04 -04:00
Daniel Nephin e03e911144 state: fix bug in changeTrackerDB.publish
Creating a new readTxn does not work because it will not see the newly created objects that are about to be committed. Instead use the active write Txn.
2020-09-03 16:58:01 -04:00
Daniel Nephin 5de4d5bbe3 stream: have SnapshotFunc accept a non-pointer SubscribeRequest
The value is not expected to be modified. Passing a value makes that explicit.
2020-09-03 16:54:02 -04:00
freddygv cd4cf5161f Update resolver defaulting 2020-09-03 13:08:44 -06:00
freddygv 00f2794bfa Update golden files after default route fix for tgw 2020-09-03 12:35:11 -06:00
Daniel Nephin 6ca45e1a61 agent: add apiServers type for managing HTTP servers
Remove Server field from HTTPServer. The field is no longer used.
2020-09-03 13:40:12 -04:00
freddygv 318aa094fd Fix http assertion in route creation 2020-09-03 10:21:20 -06:00
freddygv 30ba080d25 Add explicit protocol overrides in tgw xds test cases 2020-09-03 08:57:48 -06:00
freddygv eaa250cc80 Ensure resolver node with LB isn't considered default 2020-09-03 08:55:57 -06:00
freddygv ef877449ce Move valid policies to pkg level 2020-09-02 15:49:03 -06:00
freddygv f81fe6a1a1 Remove LB infix and move injection to xds 2020-09-02 15:13:50 -06:00
R.B. Boyer 119e945c3e
connect: all config entries pick up a meta field (#8596)
Fixes #8595
2020-09-02 14:10:25 -05:00
Chris Piraino 28f163c2d2
Merge pull request #8603 from hashicorp/feature/usage-metrics
Track node and service counts in the state store and emit them periodically as metrics
2020-09-02 13:23:39 -05:00
R.B. Boyer d0f74cd1e8
connect: fix bug in preventing some namespaced config entry modifications (#8601)
Whenever an upsert/deletion of a config entry happens, within the open
state store transaction we speculatively test compile all discovery
chains that may be affected by the pending modification to verify that
the write would not create an erroneous scenario (such as splitting
traffic to a subset that did not exist).

If a single discovery chain evaluation references two config entries
with the same kind and name in different namespaces then sometimes the
upsert/deletion would be falsely rejected. It does not appear as though
this bug would've let invalid writes through to the state store so the
correction does not require a cleanup phase.
2020-09-02 10:47:19 -05:00
Chris Piraino bcb586bee2 Set metrics reporting interval to 9 seconds
This is below the 10 second interval that lib/telemetry.go implements as
its aggregation interval, ensuring that we always report these metrics.
2020-09-02 10:24:23 -05:00
Chris Piraino a3028cad89 Update godoc string for memdb wrapper functions/structs 2020-09-02 10:24:22 -05:00
Chris Piraino d301145e62 Refactor state store usage to track unique service names
This commit refactors the state store usage code to track unique service
name changes on transaction commit. This means we only need to lookup
usage entries when reading the information, as opposed to iterating over
a large number of service indices.

- Take into account a service instance's name being changed
- Do not iterate through entire list of service instances, we only care
about whether there is 0, 1, or more than 1.
2020-09-02 10:24:21 -05:00
Chris Piraino 086a8ea8eb Use ReadTxn interface in state store helper functions 2020-09-02 10:24:20 -05:00
Chris Piraino 69dbc926ad Add WriteTxn interface and convert more functions to ReadTxn
We add a WriteTxn interface for use in updating the usage memdb table,
with the forward-looking prospect of incrementally converting other
functions to accept interfaces.

As well, we use the ReadTxn in new usage code, and as a side effect
convert a couple of existing functions to use that interface as well.
2020-09-02 10:24:19 -05:00
Chris Piraino 3feae7f77b Report node/service usage metrics from every server
Using the newly provided state store methods, we periodically emit usage
metrics from the servers.

We decided to emit these metrics from all servers, not just the leader,
because that means we do not have to care about leader election flapping
causing metrics turbulence, and it seems reasonable for each server to
emit its own view of the state, even if they should always converge
rapidly.
2020-09-02 10:24:17 -05:00
Chris Piraino 04705e90f9 Add new usage memdb table that tracks usage counts of various elements
We update the usage table on Commit() by using the TrackedChanges() API
of memdb.

Track memdb changes on restore so that usage data can be compiled
2020-09-02 10:24:16 -05:00
freddygv 63f79e5f9b Restructure structs and other PR comments 2020-09-02 09:10:50 -06:00
Daniel Nephin f1a41318d7 token: OSS support for enterprise tokens 2020-08-31 15:10:15 -04:00
Daniel Nephin 629e4aaa65 config: use token.Config for ACLToken config
Using the target Config struct reduces the amount of copying and
translating of configuration structs.
2020-08-31 15:10:15 -04:00
Daniel Nephin 330be5b740 agent/token: Move token persistence out of agent
And into token.Store. This change isolates any awareness of token
persistence in a single place.

It is a small step in allowing Agent.New to accept its dependencies.
2020-08-31 15:00:34 -04:00
Daniel Nephin a80de898ea fix TestStore_RegularTokens
This test was only passing because t.Parallel was causing every subtest to run with the last value in the iteration,
which sets a value for all tokens. The test started to fail once t.Parallel was removed, but the same failure could
have been produced by adding 'tt := tt' to the t.Run() func.

These tests run in under 10ms, so there is no reason to use t.Parallel.
2020-08-31 14:59:14 -04:00
Matt Keeler 91d680b830
Merge of auto-config and auto-encrypt code (#8523)
auto-encrypt is now handled as a special case of auto-config.

This also is moving all the cert-monitor code into the auto-config package.
2020-08-31 13:12:17 -04:00
freddygv 0236e169bb Add documentation for resolver LB cfg 2020-08-28 14:46:13 -06:00
freddygv 28d0602fc1 Pass LB config to Envoy via xDS 2020-08-28 14:27:40 -06:00
freddygv 2bbbd9e1da Log error as error 2020-08-28 13:11:55 -06:00
freddygv 81115b6eaa Compile down LB policy to disco chain nodes 2020-08-28 13:11:04 -06:00
Daniel Nephin 6956477be5
Merge pull request #8548 from edevil/fix_flake
Fix flaky TestACLResolver_Client/Concurrent-Token-Resolve
2020-08-28 15:10:55 -04:00
Daniel Nephin 72bf350069
Merge pull request #8552 from pierresouchay/reload_cache_throttling_config
Ensure that Cache options are reloaded when `consul reload` is performed
2020-08-28 15:04:42 -04:00
Pierre Souchay d5974b1d17 Added Unit test for cache reloading 2020-08-28 13:03:58 +02:00