Commit Graph

67 Commits

Author SHA1 Message Date
R.B. Boyer 6742340878
mesh: add ComputedImplicitDestinations resource for future use (#20547)
Creates a new controller to create ComputedImplicitDestinations resources by 
composing ComputedRoutes, Services, and ComputedTrafficPermissions to 
infer all ParentRef services that could possibly send some portion of traffic to a 
Service that has at least one accessible Workload Identity. A followup PR will 
rewire the sidecar controller to make use of this new resource.

As this is a performance optimization, rather than a security feature the following 
aspects of traffic permissions have been ignored:

- DENY rules
- port rules (all ports are allowed)

Also:

- Add some v2 TestController machinery to help test complex dependency mappers.
2024-02-09 15:42:10 -06:00
Eric Haberkorn b26282568f
Move sameness groups to v2beta1 version (#20531) 2024-02-08 11:05:06 -05:00
Matt Keeler 3ca4f39fa1
Register the multicluster types for the catalogtest integration tests (#20516)
In particular the failover controller needs these in Consul Enterprise
2024-02-07 13:35:02 -05:00
R.B. Boyer deca6a49bd
catalog: improve the bound workload identity encoding on services (#20458)
The endpoints controller currently encodes the list of unique workload identities 
referenced by all workload matched by a Service into a special data-bearing 
status condition on that Service. This allows a downstream controller to avoid an 
expensive watch on the ServiceEndpoints type just to get this data.

The current encoding does not lend itself well to machine parsing, which is what 
the field is meant for, so this PR simplifies the encoding from:

    "blah blah: " + strings.Join(ids, ",") + "."

to

    strings.Join(ids, ",")

It also provides an exported utility function to easily extract this data.
2024-02-02 16:28:39 -06:00
Eric Haberkorn d0243b618d
Change the multicluster group to v2 (#20430) 2024-02-01 12:08:26 -05:00
Matt Keeler 34a32d4ce5
Remove V2 PeerName field from pbresource.Tenancy (#19865)
The peer name will eventually show up elsewhere in the resource. For now though this rips it out of where we don’t want it to be.
2024-01-29 15:08:31 -05:00
Nitya Dhanushkodi 92aab7ea31
[NET-5586][rebased] v2: Support virtual port references in config (#20371)
[OG Author: michael.zalimeni@hashicorp.com, rebase needed a separate PR]

* v2: support virtual port in Service port references

In addition to Service target port references, allow users to specify a
port by stringified virtual port value. This is useful in environments
such as Kubernetes where typical configuration is written in terms of
Service virtual ports rather than workload (pod) target port names.

Retaining the option of referencing target ports by name supports VMs,
Nomad, and other use cases where virtual ports are not used by default.

To support both uses cases at once, we will strictly interpret port
references based on whether the value is numeric. See updated
`ServicePort` docs for more details.

* v2: update service ref docs for virtual port support

Update proto and generated .go files with docs reflecting virtual port
reference support.

* v2: add virtual port references to L7 topo test

Add coverage for mixed virtual and target port references to existing
test.

* update failover policy controller tests to work with computed failover policy and assert error conditions against FailoverPolicy and ComputedFailoverPolicy resources

* accumulate services; don't overwrite them in enterprise

---------

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2024-01-29 10:43:41 -08:00
aahel 3446eb3b1b
added computed failover controller (#20329)
* added computed failover controller

* removed some uncessary changes

* removed uncessary changes

* minor refactor

* minor refactor fmt

* added copyright
2024-01-24 11:50:27 +05:30
Matt Keeler 0a261682cd
Migrate the node health controller to use the cache (#20248)
Some edge case error testing had to be removed because it was no longer possible to force errors when going through the cache layer as opposed to the resource service itself.
2024-01-19 12:22:45 -05:00
Matt Keeler cee9df574d
Deflake the catalog v2beta1 integration tests (#20278) 2024-01-19 10:49:47 -05:00
Matt Keeler f9c04881f9
Failover policy cache (#20244)
* Migrate the Failover controller to use the controller cache
* Remove the Catalog FailoverMapper and its usage in the mesh routes controller.
2024-01-19 09:35:34 -05:00
Matt Keeler 59cb12c798
Migrate the Endpoints controller to use the controller cache (#20241)
* Add cache resource decoding helpers

* Implement a common package for workload selection facilities. This includes:

   * Controller cache Index
   * ACL hooks
   * Dependency Mapper to go from workload to list of resources which select it
   * Dependency Mapper to go from a resource which selects workloads to all the workloads it selects.

* Update the endpoints controller to use the cache instead of custom mappers.

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2024-01-18 17:52:52 -05:00
Matt Keeler 9897be76ad
Update workload health controller to use the controller cache (#20240) 2024-01-18 16:30:11 -05:00
Dan Stough 15ab80c832
feat(v2dns): move DNSPolicy to workload/endpoints (#20246) 2024-01-18 11:37:42 -05:00
Matt Keeler 326c0ecfbe
In-Memory gRPC (#19942)
* Implement In-Process gRPC for use by controller caching/indexing

This replaces the pipe base listener implementation we were previously using. The new style CAN avoid cloning resources which our controller caching/indexing is taking advantage of to not duplicate resource objects in memory.

To maintain safety for controllers and for them to be able to modify data they get back from the cache and the resource service, the client they are presented in their runtime will be wrapped with an autogenerated client which clones request and response messages as they pass through the client.

Another sizable change in this PR is to consolidate how server specific gRPC services get registered and managed. Before this was in a bunch of different methods and it was difficult to track down how gRPC services were registered. Now its all in one place.

* Fix race in tests

* Ensure the resource service is registered to the multiplexed handler for forwarding from client agents

* Expose peer streaming on the internal handler
2024-01-12 11:54:07 -05:00
aahel ae998a698a
added computed failover policy resource (#19975) 2023-12-18 05:52:24 +00:00
Matt Keeler 123bc95e1a
Add Common Controller Caching Infrastructure (#19767)
* Add Common Controller Caching Infrastructure
2023-12-13 10:06:39 -05:00
Semir Patel c1bbda8128
resource: block default namespace deletion + test refactorings (#19822) 2023-12-05 14:00:06 -05:00
aahel 7936e55807
added node health resource (#19803) 2023-12-02 11:14:03 +05:30
aahel ac9261ac3e
made node parition scoped (#19794)
* made node parition scoped

* removed namespace from node testdata
2023-12-01 07:42:29 +00:00
Ganesh S 2e28aecff8
Added tenancy tests for endpoints controller (#19650) 2023-11-15 21:32:26 +05:30
R.B. Boyer b21851c903
test: add test helper to detect if the current build is enterprise (#19201)
This can help conditionally change test behavior if the test is executing in enterprise without the need to split the test with build tags.
2023-11-13 10:30:07 -06:00
Kumar Kavish f09dbb99e9
[NET-6356] Add tenancy to Failover Tests (#19547)
* [NET-6356] Add tenancy to Failover Tests

* [NET-6438] Add tenancy to xDS Tests
- Added cleanup post test run

* [NET-6356] Add tenancy to failover Tests
- using t.Cleanup instead of defer delete
2023-11-10 01:14:09 +05:30
Matt Keeler a7774a9538
Introduce randomized timings and reproducible randomization into controller integration tests. (#19393)
As the V2 architecture hinges on eventual consistency and controllers reconciling the existing state in response to writes, there are potential issues we could run into regarding ordering and timing of operations. We want to be able to guarantee that given a set of resources the system will always eventually get to the desired correct state. The order of resource writes and delays in performing those writes should not alter the final outcome of reaching the desired state.

To that end, this commit introduces arbitrary randomized delays before performing resources writes into the `resourcetest.Client`. Its `PublishResources` method was already randomizing the order of resource writes. By default, no delay is added to normal writes and deletes but tests can opt-in via either passing hard coded options when creating the `resourcetest.Client` or using the `resourcetest.ConfigureTestCLIFlags` function to allow processing of CLI parameters.

In addition to allowing configurability of the request delay min and max, the client also has a configurable random number generator seed. When Using the CLI parameter helpers, a test log will be written noting the currently used settings. If the test fails then you can reproduce the same delays and order randomizations by providing the seed during the previous test failure.
2023-11-08 10:45:25 -05:00
Ashesh Vidyut 985aa76da3
NET 6354 - Add tenancy in Node Health Controller (#19457)
* node health controller tenancy

* some prog

* some fixes

* revert

* pr comment resolved

* removed name

* cleanup nodes

* some fixes

* merge main
2023-11-08 13:01:17 +05:30
Ganesh S 5352ff945c
Added tenancy tests for WorkloadHealth controller (#19530) 2023-11-07 09:09:15 +05:30
Matt Keeler 5698353652
Resource Hook Pre-Decode Utilities (#18548)
Add some generic type hook wrappers to first decode the data

There seems to be a pattern for Validation, Mutation and Write Authorization hooks where they first need to decode the Any data before doing the domain specific work.

This PR introduces 3 new functions to generate wrappers around the other hooks to pre-decode the data into a DecodedResource and pass that in instead of the original pbresource.Resource.

This PR also updates the various catalog data types to use the new hook generators.
2023-10-26 16:39:06 -04:00
Michael Zalimeni a7803bd829
[NET-6305] xds: Ensure v2 route match and protocol are populated for gRPC (#19343)
* xds: Ensure v2 route match is populated for gRPC

Similar to HTTP, ensure that route match config (which is required by
Envoy) is populated when default values are used.

Because the default matches generated for gRPC contain a single empty
`GRPCRouteMatch`, and that proto does not directly support prefix-based
config, an interpretation of the empty struct is needed to generate the
same output that the `HTTPRouteMatch` is explicitly configured to
provide in internal/mesh/internal/controllers/routes/generate.go.

* xds: Ensure protocol set for gRPC resources

Add explicit protocol in `ProxyStateTemplate` builders and validate it
is always set on clusters. This ensures that HTTP filters and
`http2_protocol_options` are populated in all the necessary places for
gRPC traffic and prevents future unintended omissions of non-TCP
protocols.

Co-authored-by: John Murret <john.murret@hashicorp.com>

---------

Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-10-25 17:43:58 +00:00
Iryna Shustava dfea3a0efe
acls,catalog,mesh: properly authorize workload selectors on writes (#19260)
To properly enforce writes on resources that have workload selectors with prefixes, we need another service authorization rule that allows us to check whether read is allowed within a given prefix. Specifically we need to only allow writes if the policy prefix allows for a wider set of names than the prefix selector on the resource. We should also not allow policies with exact names for prefix matches.

Part of [NET-3993]
2023-10-19 11:09:41 -06:00
Semir Patel 4c5a46e5e1
v2tenancy: rename v1alpha1 -> v2beta1 (#19227) 2023-10-16 21:43:47 +00:00
R.B. Boyer 6741392a4f
catalog: add FailoverPolicy ACL hook tenancy test (#19179) 2023-10-16 14:05:39 -05:00
Iryna Shustava 105ebfdd00
catalog, mesh: implement missing ACL hooks (#19143)
This change adds ACL hooks to the remaining catalog and mesh resources, excluding any computed ones. Those will for now continue using the default operator:x permissions.

It refactors a lot of the common testing functions so that they can be re-used between resources.

There are also some types that we don't yet support (e.g. virtual IPs) that this change adds ACL hooks to for future-proofing.
2023-10-13 23:16:26 +00:00
Iryna Shustava 2ea33e9b86
mesh: add more validations to Destinations resource (#19202) 2023-10-13 16:52:20 -06:00
R.B. Boyer 99f7a1219e
catalog: add metadata filtering to refine workload selectors (#19198)
This implements the Filter field on pbcatalog.WorkloadSelector to be
a post-fetch in-memory filter using the https://github.com/hashicorp/go-bexpr
expression language to filter resources based on their envelope metadata fields.

All existing usages of WorkloadSelector should be able to make use of the filter.
2023-10-13 13:37:42 -05:00
Iryna Shustava 54a12ab3c9
mesh: sidecar proxy controller improvements (#19083)
This change builds on #19043 and #19067 and updates the sidecar controller to use those computed resources. This achieves several benefits:

   * The cache is now simplified which helps us solve for previous bugs (such as multiple Upstreams/Destinations targeting the same service would overwrite each other)
   * We no longer need proxy config cache
   * We no longer need to do merging of proxy configs as part of the controller logic
   * Controller watches are simplified because we no longer need to have complex mapping using cache and can instead use the simple ReplaceType mapper.

It also makes several other improvements/refactors:

  * Unifies all caches into one. This is because originally the caches were more independent, however, now that they need to interact with each other it made sense to unify them where sidecar proxy controller uses one cache with 3 bimappers
   * Unifies cache and mappers. Mapper already needed all caches anyway and so it made sense to make the cache do the mapping also now that the cache is unified.
   * Gets rid of service endpoints watches. This was needed to get updates in a case when service's identities have changed and we need to update proxy state template's spiffe IDs for those destinations. This will however generate a lot of reconcile requests for this controller as service endpoints objects can change a lot because they contain workload's health status. This is solved by adding a status to the service object tracking "bound identities" and have service endpoints controller update it. Having service's status updated allows us to get updates in the sidecar proxy controller because it's already watching service objects
   * Add a watch for workloads. We need it so that we get updates if workload's ports change. This also ensures that we update cached identities in case workload's identity changes.
2023-10-12 13:20:13 -06:00
R.B. Boyer 29ba5b5c79
catalog: block unsupported failover policy settings for now (#19168) 2023-10-12 11:13:56 -05:00
John Murret 6da4798e05
NET-5799 - ensure catalog controllers and dependency mappers function correctly for tenancy fields (#19142)
* use bimapper

* WIP

* clean up

* PR feedback
2023-10-12 02:07:50 +00:00
Iryna Shustava c35df12c95
mesh: Add ComputedProxyConfiguration and a controller that computes it. (#19043)
* Introduce a new type `ComputedProxyConfiguration` and add a controller for it. This is needed for two reasons. The first one is that external integrations like kubernetes may need to read the fully computed and sorted proxy configuration per workload. The second reasons is that it makes sidecar-proxy controller logic quite a bit simpler as it no longer needs to do this.
* Generalize workload selection mapper and fix a bug where it would delete IDs from the tree if only one is left after a removal is done.
2023-10-10 17:34:53 -06:00
Iryna Shustava e6b724d062
catalog,mesh,auth: Move resource types to the proto-public module (#18935) 2023-09-22 15:50:56 -06:00
Iryna Shustava d88888ee8b
catalog,mesh,auth: Bump versions to v2beta1 (#18930) 2023-09-22 10:51:15 -06:00
R.B. Boyer de231bbbdd
catalog: fix for new method argument (#18978) 2023-09-22 10:42:16 -05:00
R.B. Boyer ec6189fd2f
catalog: add ACL checks for FailoverPolicy resources (#18919)
FailoverPolicy resources are name-aligned with the Service they control.
They also contain a list of possible failover destinations that are References
 to other Services.

The ACLs should be:

- list: (default)
- read: service:<resource_name>:read
- write: service:<resource_name>:write + service:<destination_name>:read (for any destination)
2023-09-22 09:59:14 -05:00
R.B. Boyer d574473fd1
mesh: make FailoverPolicy work in xdsv2 and ProxyStateTemplate (#18900)
Ensure that configuring a FailoverPolicy for a service that is reachable via a xRoute or a direct upstream causes an envoy aggregate cluster to be created for the original cluster name, but with separate clusters for each one of the possible destinations.
2023-09-20 11:59:01 -05:00
R.B. Boyer b4d5178e5c
catalog: normalize/default/validate tenancy components of FailoverPolicy internal References (#18825)
FailoverPolicy resources contain inner Reference fields. We want to ensure 
that components of those reference Tenancy fields left unspecified are defaulted 
using the tenancy of the enclosing FailoverPolicy resource.

As the underlying helper being used to do the normalization calls the function 
modified in #18822, it also means that the PeerName field will be set to "local" for 
now automatically to avoid "local" != "" issues downstream.
2023-09-18 14:59:08 -05:00
Iryna Shustava a89938e0c1
catalog: Default protocol to tcp in catalog.Service if unspecified (#18832) 2023-09-15 15:11:56 -06:00
Iryna Shustava 7f2a1d9812
catalog: service endpoints inherits protocol from service when workload doesn't have one (#18792) 2023-09-14 08:52:12 -06:00
John Murret 3e78b4cf34
Prefix sidecar proxy test files with source and destination. (#18620)
* mesh-controller: handle L4 protocols for a proxy without upstreams

* sidecar-controller: Support explicit destinations for L4 protocols and single ports.

* This controller generates and saves ProxyStateTemplate for sidecar proxies.
* It currently supports single-port L4 ports only.
* It keeps a cache of all destinations to make it easier to compute and retrieve destinations.
* It will update the status of the pbmesh.Upstreams resource if anything is invalid.

* endpoints-controller: add workload identity to the service endpoints resource

* small fixes

* review comments

* Address PR comments

* sidecar-proxy controller: Add support for transparent proxy

This currently does not support inferring destinations from intentions.

* PR review comments

* mesh-controller: handle L4 protocols for a proxy without upstreams

* sidecar-controller: Support explicit destinations for L4 protocols and single ports.

* This controller generates and saves ProxyStateTemplate for sidecar proxies.
* It currently supports single-port L4 ports only.
* It keeps a cache of all destinations to make it easier to compute and retrieve destinations.
* It will update the status of the pbmesh.Upstreams resource if anything is invalid.

* endpoints-controller: add workload identity to the service endpoints resource

* small fixes

* review comments

* Make sure endpoint refs route to mesh port instead of an app port

* Address PR comments

* fixing copyright

* tidy imports

* sidecar-proxy controller: Add support for transparent proxy

This currently does not support inferring destinations from intentions.

* tidy imports

* add copyright headers

* Prefix sidecar proxy test files with source and destination.

* Update controller_test.go

---------

Co-authored-by: Iryna Shustava <iryna@hashicorp.com>
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>
2023-09-08 17:28:55 -06:00
Iryna Shustava 3c70e14713
sidecar-proxy controller: L4 controller with explicit upstreams (NET-3988) (#18352)
* This controller generates and saves ProxyStateTemplate for sidecar proxies.
* It currently supports single-port L4 ports only.
* It keeps a cache of all destinations to make it easier to compute and retrieve destinations.
* It will update the status of the pbmesh.Upstreams resource if anything is invalid.
* This commit also changes service endpoints to include workload identity. This made the implementation a bit easier as we don't need to look up as many workloads and instead rely on endpoints data.
2023-09-07 09:37:15 -06:00
Semir Patel b96cff7436
resource: Require scope for resource registration (#18635) 2023-09-01 09:44:53 -05:00
Michael Zalimeni 699aa47416
fix: make UNSPECIFIED protocol pass validation (#18634)
We explicitly enumerate the allowed protocols in validation, so this
change is necessary to use the new enum value.

Also add tests for enum validators to ensure they stay aligned to
protos unless we explicitly want them to diverge.
2023-08-31 19:59:29 +00:00