2523 Commits

Author SHA1 Message Date
R.B. Boyer
a2a8e9c783
connect: intentions are now managed as a new config entry kind "service-intentions" (#8834)
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
Daniel Nephin
5972bdc87c streaming: improve godoc for cache-type
And fix a bug where any error that implemented the temporary interface was considered
a temporary error, even when the method would return false.
2020-10-06 13:52:02 -04:00
Daniel Nephin
d0b87cd5d0 submatview: remove notifyUpdateLocked from reset
This call appears to only be necessary because reset() was called from
NewMaterializer.

This commit has the constructor set a default value for updateCh, and
removes both the call to reset() from New(), and the call to
notifyUpdateLocked() from reset().

This should ensure that we do not notify the Fetch() call before we have new
values to report.
2020-10-06 13:22:02 -04:00
Daniel Nephin
5eab1d8cef submatview: remove method receiver from handlers 2020-10-06 13:22:02 -04:00
Daniel Nephin
3fa08beecf submatview: add a test for handling of NewSnapshotToFollow
Also add some godoc
Rename some vars and functions
Fix a data race in the new cache test for entry closing.
2020-10-06 13:22:02 -04:00
Daniel Nephin
534d8b45bb submatview: refactor Materializer
Refactor of Materializer.Run
Use handlers to manage state in Materializer
Rename Materializer receiver
rename m.l to m.lock, and flip some conditionals to remove the negative.
Improve godoc, rename Deps, move resetErr, and pass err into notifyUpdate
Update for NewSnapshotToFollow events
Refactor to move context cancel out of Materializer
2020-10-06 13:22:02 -04:00
Daniel Nephin
e849f6d7ac submatview: Move the 'use materialize from result.State' logic
No need to do all this other work if we have one already.

This logic moved closer to this call site 3 times during the process
of refactoring.
2020-10-06 13:22:02 -04:00
Daniel Nephin
edf30b2714 submatview: Move Materializer to submatview package 2020-10-06 13:22:02 -04:00
Daniel Nephin
ed45957ffb submatview: Refactor MaterializeView
Replace InitFilter with Reset.
Removes the need to store a fatalErr and the cache-type, and removes the need to recreate the filter
each time.
Pass dependencies into MaterializedView.
Remove context from MaterializedView.
Rename state to view.
Rename MaterialziedView to Materialzier.
Rename to NewMaterializer
Pass in retry.Waiter
2020-10-06 13:22:02 -04:00
Daniel Nephin
b576a2d3c7 cache-types: Update Streaming health cache-type
To use latest protobuf types
2020-10-06 13:22:02 -04:00
Daniel Nephin
132b76acef agent/cache: Add cache-type and materialized view for streaming health
Extracted from d97412ce4c399a35b41bbdae2716f0e32dce80bf

Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 13:21:57 -04:00
Daniel Nephin
b27068b72a stream: Return a single event from a subscription.Next
Handle batch events as a single event
2020-10-06 13:18:20 -04:00
Daniel Nephin
f5d11562f2 subscribe: update to use NewSnapshotToFollow event 2020-10-06 12:49:35 -04:00
Daniel Nephin
e3290f5971 Move agent/subscribe -> agent/rpc/subscribe 2020-10-06 12:49:35 -04:00
Daniel Nephin
dbb8bd679f subscirbe: extract streamID and logging from Subscribe
By extracting all of the tracing logic the core logic of the Subscribe
endpoint is much easier to read.
2020-10-06 12:49:35 -04:00
Daniel Nephin
9e4ebacb05 subscribe: add integration test for acl token updates 2020-10-06 12:49:35 -04:00
Daniel Nephin
39beed0af6 subscribe: add integration test for filtering events by acl 2020-10-06 12:49:35 -04:00
Daniel Nephin
083f4e8f57 subscribe: Add an integration test for forward to DC 2020-10-06 12:49:35 -04:00
Daniel Nephin
013ababda4 subscribe: add first integration test for Server 2020-10-06 12:49:35 -04:00
Daniel Nephin
d0256a0c07 subscribe: add a stateless subscribe service for the gRPC server
With a Backend that provides access to the necessary dependencies.
2020-10-06 12:49:35 -04:00
Daniel Nephin
f4ea3066fb subscribe: add commented out test cases
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin
106d781dc9 subscribe: add initial impl from streaming-rpc-final branch
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin
364f6589c8
Merge pull request #8799 from hashicorp/streaming/rename-framing-events
stream: remove EndOfEmptySnapshot, add NewSnapshotToFollow
2020-10-06 12:42:58 -04:00
Daniel Nephin
a5c50c982d
Merge pull request #8806 from hashicorp/dnephin/service-health-interface
rpcclient: Add health.Client and use it in http and dns
2020-10-06 12:13:09 -04:00
R.B. Boyer
4998a08c56
server: create new memdb table for storing system metadata (#8703)
This adds a new very tiny memdb table and corresponding raft operation
for updating a very small effective map[string]string collection of
"system metadata". This can persistently record a fact about the Consul
state machine itself.

The first use of this feature will come in a later PR.
2020-10-06 10:08:37 -05:00
hashicorp-ci
356998d91e auto-updated agent/uiserver/bindata_assetfs.go from commit a7038b48c 2020-10-06 13:43:27 +00:00
hashicorp-ci
bc5ca8b638 auto-updated agent/uiserver/bindata_assetfs.go from commit 5a39be47d 2020-10-06 13:32:27 +00:00
hashicorp-ci
36d219e2cc auto-updated agent/uiserver/bindata_assetfs.go from commit 76d3909f3 2020-10-06 08:42:39 +00:00
hashicorp-ci
4c9630ffad auto-updated agent/uiserver/bindata_assetfs.go from commit e4a0dcf10 2020-10-06 08:36:18 +00:00
Freddy
22062ba9da
Add default meta to test assertion (#8824) 2020-10-05 15:00:25 -06:00
Daniel Nephin
5a5fd4f0b1
Merge pull request #8802 from hashicorp/dnephin/extract-lib-retry
lib/retry - extract a new package from lib/retry.go
2020-10-05 14:22:37 -04:00
hashicorp-ci
3e9bd96c0a auto-updated agent/uiserver/bindata_assetfs.go from commit f26201a7a 2020-10-05 17:12:56 +00:00
freddygv
413a894a1a Do not evaluate discovery chain for topology upstreams 2020-10-05 10:24:50 -06:00
freddygv
cf7b7fcdd6 Single DB txn for ServiceTopology and other PR comments 2020-10-05 10:24:50 -06:00
freddygv
7c26a71b4b Add topology HTTP endpoint 2020-10-05 10:24:50 -06:00
freddygv
dbbf6b2e46 Add topology RPC endpoint 2020-10-05 10:24:50 -06:00
freddygv
98c81976f5 Add topology ACL filter 2020-10-05 10:24:50 -06:00
freddygv
f906b94351 Add func to combine up+downstream queries 2020-10-05 10:24:50 -06:00
freddygv
5c913ec312 factor in discovery chain when querying up/downstreams 2020-10-05 10:24:50 -06:00
freddygv
b012d8374e support querying upstreams/downstreams from registrations 2020-10-05 10:24:50 -06:00
freddygv
a86cf88a4a Add method for downstreams from disco chain 2020-10-05 10:24:50 -06:00
hashicorp-ci
896d143647 auto-updated agent/uiserver/bindata_assetfs.go from commit 3a89c7d23 2020-10-05 14:07:13 +00:00
hashicorp-ci
bf2b52f880 auto-updated agent/uiserver/bindata_assetfs.go from commit dfe57406d 2020-10-05 13:53:36 +00:00
hashicorp-ci
81baad4038 auto-updated agent/uiserver/bindata_assetfs.go from commit b871837ee 2020-10-05 13:14:40 +00:00
Daniel Nephin
529f252d5c rpcclient: Add health.Client and use it in http and dns
This new package provides a client agent implementation of an interface
for fetching the health of services.

This approach has a number of benefits:

1. It provides a much more explicit interface. Instead of everything
   dependency on `RPC()` and `Cache.Get()` for many unrelated things
   they can depend on a type that are named according to the behaviour
   it provides.

2. It gives us a single place to vary the behaviour and migrate to
   a new form of RPC (gRPC). The current implementation has two options
   (cache, or direct RPC), and in the future we will have more.
   It is also a great opporunity to start adding `context.Context` args
   to these operations, which in the future will allow us to cancel
   the operations.

3. As a concequence of the first, in the Server agent where we make
   these calls we can replace the current in-memory RPC calls with
   a thin adapter for the real method. This removes the `net/rpc`
   machinery from the call in places where it is not needed.

This new package is quite small right now, but I think we can expect it
to grow to a more reasonable size as other RPC calls are replaced.

This change also happens to replace two very similar implementations with
a single implementation.
2020-10-04 18:55:02 -04:00
Daniel Nephin
e54567223b lib/retry: Refactor to reduce the interface surface
Reduce Jitter to one function

Rename NewRetryWaiter

Fix a bug in calculateWait where maxWait was applied before jitter, which would make it
possible to wait longer than maxWait.
2020-10-04 18:12:42 -04:00
Daniel Nephin
ca26dfb4a2 lib/retry: extract a new package from lib 2020-10-04 17:43:01 -04:00
Kit Patella
f5c51ae13b remove consul.api.http from filtered metric prefixes 2020-10-02 14:16:02 -07:00
Kit Patella
52451cf846
Merge pull request #8271 from coignetp/http-metrics-label
Use method and path as labels for http metrics
2020-10-02 13:41:48 -07:00
hashicorp-ci
90b5f1a838 auto-updated agent/uiserver/bindata_assetfs.go from commit 8b409529a 2020-10-02 19:28:38 +00:00