Commit Graph

2597 Commits

Author SHA1 Message Date
Daniel Nephin c4122edd22 cache: export ExpiryHeap
and hide internal methods on an unexported type, so that when it is extrated those methods are not exported.
2020-10-20 19:10:20 -04:00
Daniel Nephin 343d133183 cache: Refactor heap.notify to make it more explicit.
And remove duplicate notifications.

Instead of performing the check in the heap implementation, check the
index in the higher level interface (Add,Remove,Update) and notify if one
of the relevant indexes is 0.
2020-10-20 19:10:20 -04:00
Daniel Nephin 499f2822cf cache: Move more of the expiryLoop into the Heap 2020-10-20 19:10:20 -04:00
Daniel Nephin 2cdc90e01b cache: extract cache eviction heap
Start creating an interface that doesn't require using heap and hides more of the
entry internals.
2020-10-20 19:10:19 -04:00
Daniel Nephin 3c8929c7e1 streaming: apply filter to a single item
Instead of the whole map. This should save a lot of time performing reflecting on a large map.
The filter does not change, so there is no reason to re-apply it to older entries.
2020-10-19 18:24:02 -04:00
hashicorp-ci 6b68e1c997 auto-updated agent/uiserver/bindata_assetfs.go from commit 2773daedc 2020-10-19 16:35:48 +00:00
hashicorp-ci 13c06ba8bc auto-updated agent/uiserver/bindata_assetfs.go from commit b24c2227d 2020-10-19 16:19:40 +00:00
hashicorp-ci 6513faf14c auto-updated agent/uiserver/bindata_assetfs.go from commit 02dcd422f 2020-10-19 14:50:52 +00:00
hashicorp-ci 8ff996145a auto-updated agent/uiserver/bindata_assetfs.go from commit bf8bb7415 2020-10-19 14:25:54 +00:00
Daniel Nephin 0003720f78 agent/router: refactor calculation of delay between rebalances.
This change attempts to make the delay logic more obvious by:

* remove indirection, inline a bunch of function calls
* move all the code and constants next to each other
* replace the two constant values with a single value
* reword the comments.
2020-10-15 15:59:36 -04:00
Daniel Nephin 119c446cf2 agent/router: Add bounds test cases 2020-10-15 14:43:29 -04:00
Daniel Nephin 12e174900b router: organize the test by number of servers
And adddd some additional cases to show where the minimum value stops being used
2020-10-15 13:53:37 -04:00
Daniel Nephin 8697cc2b45 router: make refreshServerRebalanceTimer test a lot more strict 2020-10-15 12:05:07 -04:00
hashicorp-ci 40c16f6ee4 auto-updated agent/uiserver/bindata_assetfs.go from commit d08e13e5b 2020-10-15 14:49:01 +00:00
hashicorp-ci 55c89133ba auto-updated agent/uiserver/bindata_assetfs.go from commit 5fd79ba40 2020-10-14 08:07:29 +00:00
Mathilde Gilles 1c8369b3c3 Fix: service LocallyRegisteredAsSidecar property is not persisted
When a service is deregistered, we check whever matching services were
registered as sidecar along with it and deregister them as well.
To determine if a service is indeed a sidecar we check the
structs.ServiceNode.LocallyRegisteredAsSidecar property. However, to
avoid interal API leakage, it is excluded from JSON serialization,
meaning it is not persisted to disk either.
When the agent is restarted, this property lost and sidecars are no
longer deregistered along with their parent service.
To fix this, we now specifically save this property in the persisted
service file.
2020-10-13 19:38:58 +02:00
hashicorp-ci 5ede07e0e7 auto-updated agent/uiserver/bindata_assetfs.go from commit 7afcd7755 2020-10-13 16:43:31 +00:00
Mike Morris 9455805ceb chore: regenerate bindata_assetfs 2020-10-12 15:44:33 -04:00
Mike Morris f3f6e57ed6 Merge branch 'master' into release/1.9.0-beta1 2020-10-12 15:32:58 -04:00
hashicorp-ci 700c163ed3
update bindata_assetfs.go 2020-10-12 18:30:30 +00:00
hashicorp-ci 564ab4146e auto-updated agent/uiserver/bindata_assetfs.go from commit 868cd47a6 2020-10-12 16:28:44 +00:00
Pierre Souchay 9b7ed75552
Consul Service meta wrongly computes and exposes non_voter meta (#8731)
* Consul Service meta wrongly computes and exposes non_voter meta

In Serf Tags, entreprise members being non-voters use the tag
`nonvoter=1`, not `non_voter = false`, so non-voters in members
were wrongly displayed as voter.

Demonstration:

```
consul members -detailed|grep voter
consul20-hk5 10.200.100.110:8301   alive   acls=1,build=1.8.4+ent,dc=hk5,expect=3,ft_fs=1,ft_ns=1,id=xxxxxxxx-5629-08f2-3a79-10a1ab3849d5,nonvoter=1,port=8300,raft_vsn=3,role=consul,segment=<all>,use_tls=1,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302
```

* Added changelog

* Added changelog entry
2020-10-09 17:18:24 -04:00
hashicorp-ci b73dea7942 auto-updated agent/uiserver/bindata_assetfs.go from commit 851705e93 2020-10-09 21:01:02 +00:00
hashicorp-ci 4f00a02737 auto-updated agent/uiserver/bindata_assetfs.go from commit 27048a061 2020-10-09 20:35:59 +00:00
Paul Banks 27048a0612
Add metrics rendering to the new topology view. (#8858)
* Remove unused StatsCard component

* Create Card and Stats contextual components with styling

* Send endpoint, item, and protocol to Stats as props

* WIP basic plumbing for metrics in Ember

* WIP metrics data source now works for different protocols and produces reasonable mock responses

* WIP sparkline component

* Mostly working metrics and graphs in topology

* Fix date in tooltip to actually be correct

* Clean up console.log

* Add loading frame and create a style sheet for Stats

* Various polish fixes:

 - Loading state for graph
 - Added fake latency cookie value to test loading
 - If metrics provider has no series/stats for the service show something that doesn't look broken
 - Graph hover works right to the edge now
 - Stats boxes now wrap so they are either shown or not as will fit not cut off
 - Graph resizes when browser window size changes
 - Some tweaks to number formats and stat metrics to make them more compact/useful

* Thread Protocol through topology model correctly

* Rebuild assetfs

* Fix failing tests and remove stats-card now it's changed and become different

* Fix merge conflict

* Update api doublt

* more merge fixes

* Add data-permission and id attr to Card

* Run JS linter

* Move things around so the tests run with everything available

* Get tests passing:

1. Remove fakeLatency setTimeout (will be replaced with CONSUL_LATENCY
in mocks)
2. Make sure any event handlers are removed

* Make sure the Consul/scripts are available before the app

* Make sure interval gets set if there is no cookie value

* Upgrade mocks so we can use CONSUL_LATENCY

* Fix handling of no series values from Prometheus

* Update assetfs and fix a comment

* Rebase and rebuild assetfs; fix tcp metric series units to be bits not bytes

* Rebuild assetfs

* Hide stats when provider is not configured

Co-authored-by: kenia <keniavalladarez@gmail.com>
Co-authored-by: John Cowen <jcowen@hashicorp.com>
2020-10-09 21:31:15 +01:00
s-christoff 9bb348c6c7
Enhance the output of consul snapshot inspect (#8787) 2020-10-09 14:57:29 -05:00
Kyle Havlovitz ff12fc9f38 Stop intermediate renew routine on leader stop 2020-10-09 12:30:57 -07:00
Kyle Havlovitz e5ab1b45bc
Merge pull request #8784 from hashicorp/renew-intermediate-primary
connect: Enable renewing the intermediate cert in the primary DC
2020-10-09 12:18:59 -07:00
hashicorp-ci 45c194f21b auto-updated agent/uiserver/bindata_assetfs.go from commit 766b28b8e 2020-10-09 18:44:15 +00:00
Daniel Nephin ea77eccb14
Merge pull request #8825 from hashicorp/streaming/add-config
streaming: add config and docs
2020-10-09 14:33:58 -04:00
Daniel Nephin e7d505dc33 config: add field for enabling streaming in the client
agent: register the new streaming cache-type
2020-10-09 14:11:34 -04:00
Daniel Nephin 0d653b184b
Merge pull request #8893 from hashicorp/dnephin/add-steps-to-subscribe-tests
subscribe: add steps to long test cases, and add new cases for converting Events
2020-10-09 13:54:59 -04:00
Daniel Nephin 8e8ba6301d
Merge pull request #8896 from hashicorp/dnephin/go-test-race-more-pkgs
ci: go test -race more packages
2020-10-09 13:48:56 -04:00
Chris Piraino 30540e406b
Emit service usage metrics with correct labeling strategy (#8856)
Previously, we would emit service usage metrics both with and without a
namespace label attached. This is problematic in the case when you want
to aggregate metrics together, i.e. "sum(consul.state.services)". This
would cause services to be counted twice in that aggregate, once via the
metric emitted with a namespace label, and once in the metric emited
without any namespace label.
2020-10-09 11:01:45 -05:00
Kyle Havlovitz 876500e0dc Fix intermediate refresh test comments 2020-10-09 08:53:33 -07:00
Matt Keeler 0d8bf9c62c
Remove oss/ent divergence of runtime_test.go (#8903) 2020-10-09 10:45:11 -04:00
Matt Keeler 8f890bc027
Add capability for the v1/connect/ca/roots endpoint to return a PEM encoded certificate chain (#8774)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-09 10:43:33 -04:00
R.B. Boyer e113dc0fe2
upstream some differences from enterprise (#8902) 2020-10-09 09:42:53 -05:00
hashicorp-ci 19abd50412 auto-updated agent/uiserver/bindata_assetfs.go from commit b6f686fec 2020-10-09 13:37:35 +00:00
R.B. Boyer b6f686fecb
uiserver: upstream refactors done elsewhere (#8891) 2020-10-09 08:32:39 -05:00
Kyle Havlovitz 01ce9f5b18 Update CI for leader renew CA test using Vault 2020-10-09 05:48:15 -07:00
Kyle Havlovitz 4fc0f6d9a4
Merge branch 'master' into renew-intermediate-primary 2020-10-09 04:40:34 -07:00
Kyle Havlovitz e13f4af06b connect: Check for expired root cert when cross-signing 2020-10-09 04:35:56 -07:00
Paul Banks f11b759ddf
Actually proxy the query string too 2020-10-09 12:27:24 +01:00
Daniel Nephin 3ff6c5b3d3 cache-types: skip tests with races 2020-10-08 20:15:13 -04:00
Daniel Nephin af8a617797 grpc: fix data rate in stats handler test 2020-10-08 19:43:49 -04:00
Freddy 13df5d5bf8
Add protocol to the topology endpoint response (#8868) 2020-10-08 17:31:54 -06:00
Daniel Nephin c5d57c9f07 subscribe: add test cases for newEventFromStreamEvent 2020-10-08 18:48:17 -04:00
Kit Patella adeabf2399
Merge pull request #8877 from hashicorp/mkcp/telemetry/consul.api.http
Add flag for disabling 1.9 metrics backwards compatibility and warnings when set to default
2020-10-08 13:22:37 -07:00
R.B. Boyer 46a4ef7248
agent: allow the /v1/connect/intentions/match endpoint to use the agent cache (#8875)
This is the recommended proxy integration API for listing intentions
which should not require an active connection to the servers to resolve
after the initial cache filling.
2020-10-08 14:51:53 -05:00
Daniel Nephin f185124320 subscribe: Add steps to rpc/subscribe tests
To make them easier to follow
2020-10-08 15:38:01 -04:00
hashicorp-ci de99c78f14 auto-updated agent/uiserver/bindata_assetfs.go from commit 13dfde75a 2020-10-08 19:36:48 +00:00
Matt Keeler 38f5ddce2a
Add per-agent reconnect timeouts (#8781)
This allows for client agent to be run in a more stateless manner where they may be abruptly terminated and not expected to come back. If advertising a per-agent reconnect timeout using the advertise_reconnect_timeout configuration when that agent leaves, other agents will wait only that amount of time for the agent to come back before reaping it.

This has the advantageous side effect of causing servers to deregister the node/services/checks for that agent sooner than if the global reconnect_timeout was used.
2020-10-08 15:02:19 -04:00
Paul Banks 332477f24c
Add /v1/internal/ui/metrics-proxy API endpoint that proxies to a configured metrics provider backend. 2020-10-08 17:32:29 +01:00
Paul Banks 80910b46f3
Fix merge conflicts 2020-10-08 17:31:36 +01:00
Daniel Nephin b93577c94f config: add field for enabling streaming RPC endpoint 2020-10-08 12:11:20 -04:00
Daniel Nephin b5b790d4c0 streaming: Use a shorter LastGetTTL for the cache 2020-10-08 12:11:20 -04:00
hashicorp-ci fc6a7df5f6 auto-updated agent/uiserver/bindata_assetfs.go from commit b373456c7 2020-10-08 15:56:05 +00:00
Freddy 164ce57db2
Support ingress gateways in mesh viz endpoint (#8864)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-08 09:47:09 -06:00
hashicorp-ci 418fe6ba32 auto-updated agent/uiserver/bindata_assetfs.go from commit ef7b1f8a4 2020-10-08 15:41:40 +00:00
hashicorp-ci 12e735dbca auto-updated agent/uiserver/bindata_assetfs.go from commit d849f025c 2020-10-08 15:08:16 +00:00
Daniel Nephin fcaf362c9f
Merge pull request #8841 from hashicorp/streaming/materialize-view-fix-service-unique-id
streaming: Use an ID that includes namespace to store services in the materialized view
2020-10-07 21:28:53 -04:00
Daniel Nephin 2513f42c68
Merge pull request #8809 from hashicorp/streaming/materialize-view
Add StreamingHealthServices cache-type
2020-10-07 21:26:38 -04:00
Daniel Nephin b103568e98
Merge pull request #8818 from hashicorp/streaming/add-subscribe-service-batch-events
stream: handle batch events as a special case of Event
2020-10-07 21:25:32 -04:00
Daniel Nephin da6400192b
Merge pull request #8768 from hashicorp/streaming/add-subscribe-service
subscribe: add subscribe service for streaming change events
2020-10-07 21:24:03 -04:00
Freddy da91e999f6
Return intention info in svc topology endpoint (#8853) 2020-10-07 18:35:34 -06:00
Kit Patella 7fe2f80b4b add config flag to disable 1.9 metrics backwards compatibility. Add warnings on start and reload on default value 2020-10-07 17:12:52 -07:00
Daniel Nephin 21c21191f4 structs: add CheckServiceNode.CanRead
And use it from the subscribe endpoint.
2020-10-07 18:15:13 -04:00
hashicorp-ci 8655b24639 auto-updated agent/uiserver/bindata_assetfs.go from commit 0d0de4309 2020-10-07 14:13:29 +00:00
hashicorp-ci eb4bc23a03 auto-updated agent/uiserver/bindata_assetfs.go from commit a61b6c53d 2020-10-07 13:50:39 +00:00
hashicorp-ci aa0f5ff839 auto-updated agent/uiserver/bindata_assetfs.go from commit 222555c4c 2020-10-07 08:09:54 +00:00
R.B. Boyer 1b413b0444
connect: support defining intentions using layer 7 criteria (#8839)
Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).
2020-10-06 17:09:13 -05:00
Daniel Nephin 8b887af0d3 streaming: store services with a unique ID that includes namespace 2020-10-06 16:54:56 -04:00
R.B. Boyer a2a8e9c783
connect: intentions are now managed as a new config entry kind "service-intentions" (#8834)
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
Daniel Nephin 5972bdc87c streaming: improve godoc for cache-type
And fix a bug where any error that implemented the temporary interface was considered
a temporary error, even when the method would return false.
2020-10-06 13:52:02 -04:00
Daniel Nephin d0b87cd5d0 submatview: remove notifyUpdateLocked from reset
This call appears to only be necessary because reset() was called from
NewMaterializer.

This commit has the constructor set a default value for updateCh, and
removes both the call to reset() from New(), and the call to
notifyUpdateLocked() from reset().

This should ensure that we do not notify the Fetch() call before we have new
values to report.
2020-10-06 13:22:02 -04:00
Daniel Nephin 5eab1d8cef submatview: remove method receiver from handlers 2020-10-06 13:22:02 -04:00
Daniel Nephin 3fa08beecf submatview: add a test for handling of NewSnapshotToFollow
Also add some godoc
Rename some vars and functions
Fix a data race in the new cache test for entry closing.
2020-10-06 13:22:02 -04:00
Daniel Nephin 534d8b45bb submatview: refactor Materializer
Refactor of Materializer.Run
Use handlers to manage state in Materializer
Rename Materializer receiver
rename m.l to m.lock, and flip some conditionals to remove the negative.
Improve godoc, rename Deps, move resetErr, and pass err into notifyUpdate
Update for NewSnapshotToFollow events
Refactor to move context cancel out of Materializer
2020-10-06 13:22:02 -04:00
Daniel Nephin e849f6d7ac submatview: Move the 'use materialize from result.State' logic
No need to do all this other work if we have one already.

This logic moved closer to this call site 3 times during the process
of refactoring.
2020-10-06 13:22:02 -04:00
Daniel Nephin edf30b2714 submatview: Move Materializer to submatview package 2020-10-06 13:22:02 -04:00
Daniel Nephin ed45957ffb submatview: Refactor MaterializeView
Replace InitFilter with Reset.
Removes the need to store a fatalErr and the cache-type, and removes the need to recreate the filter
each time.
Pass dependencies into MaterializedView.
Remove context from MaterializedView.
Rename state to view.
Rename MaterialziedView to Materialzier.
Rename to NewMaterializer
Pass in retry.Waiter
2020-10-06 13:22:02 -04:00
Daniel Nephin b576a2d3c7 cache-types: Update Streaming health cache-type
To use latest protobuf types
2020-10-06 13:22:02 -04:00
Daniel Nephin 132b76acef agent/cache: Add cache-type and materialized view for streaming health
Extracted from d97412ce4c

Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 13:21:57 -04:00
Daniel Nephin b27068b72a stream: Return a single event from a subscription.Next
Handle batch events as a single event
2020-10-06 13:18:20 -04:00
Daniel Nephin f5d11562f2 subscribe: update to use NewSnapshotToFollow event 2020-10-06 12:49:35 -04:00
Daniel Nephin e3290f5971 Move agent/subscribe -> agent/rpc/subscribe 2020-10-06 12:49:35 -04:00
Daniel Nephin dbb8bd679f subscirbe: extract streamID and logging from Subscribe
By extracting all of the tracing logic the core logic of the Subscribe
endpoint is much easier to read.
2020-10-06 12:49:35 -04:00
Daniel Nephin 9e4ebacb05 subscribe: add integration test for acl token updates 2020-10-06 12:49:35 -04:00
Daniel Nephin 39beed0af6 subscribe: add integration test for filtering events by acl 2020-10-06 12:49:35 -04:00
Daniel Nephin 083f4e8f57 subscribe: Add an integration test for forward to DC 2020-10-06 12:49:35 -04:00
Daniel Nephin 013ababda4 subscribe: add first integration test for Server 2020-10-06 12:49:35 -04:00
Daniel Nephin d0256a0c07 subscribe: add a stateless subscribe service for the gRPC server
With a Backend that provides access to the necessary dependencies.
2020-10-06 12:49:35 -04:00
Daniel Nephin f4ea3066fb subscribe: add commented out test cases
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin 106d781dc9 subscribe: add initial impl from streaming-rpc-final branch
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin 364f6589c8
Merge pull request #8799 from hashicorp/streaming/rename-framing-events
stream: remove EndOfEmptySnapshot, add NewSnapshotToFollow
2020-10-06 12:42:58 -04:00
Daniel Nephin a5c50c982d
Merge pull request #8806 from hashicorp/dnephin/service-health-interface
rpcclient: Add health.Client and use it in http and dns
2020-10-06 12:13:09 -04:00
R.B. Boyer 4998a08c56
server: create new memdb table for storing system metadata (#8703)
This adds a new very tiny memdb table and corresponding raft operation
for updating a very small effective map[string]string collection of
"system metadata". This can persistently record a fact about the Consul
state machine itself.

The first use of this feature will come in a later PR.
2020-10-06 10:08:37 -05:00
hashicorp-ci 356998d91e auto-updated agent/uiserver/bindata_assetfs.go from commit a7038b48c 2020-10-06 13:43:27 +00:00
hashicorp-ci bc5ca8b638 auto-updated agent/uiserver/bindata_assetfs.go from commit 5a39be47d 2020-10-06 13:32:27 +00:00