Commit Graph

85 Commits

Author SHA1 Message Date
Semir Patel 63cc037110
resource: Make resource read tenancy aware (#18397) 2023-08-07 16:37:03 -05:00
R.B. Boyer 1ebd001a07
bimapper: fix a bug and add some more test coverage (#18387) 2023-08-04 16:45:10 -05:00
wangxinyi7 1f28ac2664
expose grpc as http endpoint (#18221)
expose resource grpc endpoints as http endpoints
2023-08-04 11:27:48 -07:00
R.B. Boyer 9c227e2c36
mesh: adding the protobuf types and resources backing mesh config v2 (#18351) 2023-08-03 13:42:04 -05:00
Iryna Shustava a33001f4d4
Register ProxyStateTemplate Resource (#18316)
Also, change the ProxyState.id to identity. This is because we already have the id of this proxy
  from the resource, and this id should be name-aligned with the workload it represents. It should
  also have the owner ref set to the workload ID if we need that. And so the id field seems unnecessary.
  We do, however, need a reference to workload identity so that we can authorize the proxy when it initially
  connects to the xDS server.
2023-08-02 08:15:13 -06:00
R.B. Boyer 13ce787a3f
resource: adding various helpers for working with resources (#18342)
This is a bit of a grab bag of helpers that I found useful for working with them when authoring substantial Controllers. Subsequent PRs will make use of them.
2023-08-01 13:39:15 -05:00
Semir Patel efb45fe851
resource: Add scope to resource type registration [NET-4976] (#18214)
Enables querying a resource type's registration to determine if a resource is cluster, partition, or partition and namespace scoped.
2023-07-24 11:34:30 -05:00
Semir Patel ada767fc9f
resource: Pass resource to Write ACL hook instead of just resource Id [NET-4908] (#18192) 2023-07-20 12:06:29 -05:00
Semir Patel 003370ded0
Call resource mutate hook before validate hook (NET-4907) (#18178) 2023-07-19 13:10:57 -05:00
Iryna Shustava c328ba85bd
Split pbmesh.UpstreamsConfiguration as a resource out of pbmesh.Upstreams (#17991)
Configuration that previously was inlined into the Upstreams resource
applies to both explicit and implicit upstreams and so it makes sense to
split it out into its own resource.

It also has other minor changes:
- Renames `proxy.proto` proxy_configuration.proto`
- Changes the type of `Upstream.destination_ref` from `pbresource.ID` to
`pbresource.Reference`
- Adds comments to fields that didn't have them
2023-07-13 13:06:56 -06:00
Dan Upton b117eb0126
resource: enforce consistent naming of resource types (#17611)
For consistency, resource type names must follow these rules:

- `Group` must be snake case, and in most cases a single word.
- `GroupVersion` must be lowercase, start with a "v" and end with a number.
- `Kind` must be pascal case.

These were chosen because they map to our protobuf type naming
conventions.
2023-06-26 13:25:14 +01:00
Dan Upton 48445dfa55
resource: add `AuthorizerContext` helper method (#17393) 2023-06-26 13:23:05 +01:00
Matt Keeler 37636eab71
Catalog V2 Container Based Integration Test (#17674)
* Implement the Catalog V2 controller integration container tests

This now allows the container tests to import things from the root module. However for now we want to be very restrictive about which packages we allow importing.

* Add an upgrade test for the new catalog

Currently this should be dormant and not executed. However its put in place to detect breaking changes in the future and show an example of how to do an upgrade test with integration tests structured like catalog v2.

* Make testutil.Retry capable of performing cleanup operations

These cleanup operations are executed after each retry attempt.

* Move TestContext to taking an interface instead of a concrete testing.T

This allows this to be used on a retry.R or generally anything that meets the interface.

* Move to using TestContext instead of background contexts

Also this forces all test methods to implement the Cleanup method now instead of that being an optional interface.


Co-authored-by: Daniel Upton <daniel@floppy.co>
2023-06-16 16:29:50 -04:00
Matt Keeler 653a886689
Implement a Catalog Controllers Lifecycle Integration Test (#17435)
* Implement a Catalog Controllers Lifecycle Integration Test

* Prevent triggering the race detector.

This allows defining some variables for protobuf constants and using those in comparisons. Without that, something internal in the fmt package ended up looking at the protobuf message size cache and triggering the race detector.
2023-06-16 12:58:53 -04:00
Matt Keeler a5ba889034
Implement the service endpoints controller (#17216)
* Add a ReplaceType dep mapper and move them into their own file

* Implement the service endpoints controller

* Implement a Catalog Controllers Integration Test
2023-06-06 17:09:48 -04:00
Matt Keeler 77f44fa878
Various bits of cleanup detected when using Go Workspaces (#17462)
TLDR with many modules the versions included in each diverged quite a bit. Attempting to use Go Workspaces produces a bunch of errors.

This commit:

1. Fixes envoy-library-references.sh to work again
2. Ensures we are pulling in go-control-plane@v0.11.0 everywhere (previously it was at that version in some modules and others were much older)
3. Remove one usage of golang/protobuf that caused us to have a direct dependency on it.
4. Remove deprecated usage of the Endpoint field in the grpc resolver.Target struct. The current version of grpc (v1.55.0) has removed that field and recommended replacement with URL.Opaque and calls to the Endpoint() func when needing to consume the previous field.
4. `go work init <all the paths to go.mod files>` && `go work sync`. This syncrhonized versions of dependencies from the main workspace/root module to all submodules
5. Updated .gitignore to ignore the go.work and go.work.sum files. This seems to be standard practice at the moment.
6. Update doc comments in protoc-gen-consul-rate-limit to be go fmt compatible
7. Upgraded makefile infra to perform linting, testing and go mod tidy on all modules in a flexible manner.
8. Updated linter rules to prevent usage of golang/protobuf
9. Updated a leader peering test to account for an extra colon in a grpc error message.
2023-06-05 16:08:39 -04:00
Dan Stough d935c7b466
[OSS] gRPC Blocking Queries (#17426)
* feat: initial grpc blocking queries

* changelog and docs update
2023-05-23 17:29:10 -04:00
Matt Keeler 1d6a0c8f21
Add the workload health controller (#17215) 2023-05-19 13:53:29 -04:00
R.B. Boyer 21c6e0e8e6
fix two typos (#17389) 2023-05-17 08:50:26 -07:00
Matt Keeler d37572bd44
Add a Node health controller (#17214)
This will aggregate all HealthStatus objects owned by the Node and update the status of the Node with an overall health.
2023-05-15 09:55:03 -04:00
Dan Upton 879b775459
docs: initial documentation for the new State Store (#17315) 2023-05-15 12:34:36 +01:00
Matt Keeler 456156ebec
Add type validations for the catalog resources (#17211)
Also adding some common resource validation error types to the internal/resource package.
2023-05-12 09:24:55 -04:00
Dan Upton 6c24a66f73
resource: optionally compare timestamps in `EqualStatus` (#17275) 2023-05-10 10:37:54 +01:00
Semir Patel 40eefaba18
Reaper controller for cascading deletes of owner resources (#17256) 2023-05-09 13:57:40 -05:00
Dan Upton d53a1d4a27
resource: add helpers for more efficiently comparing IDs etc (#17224) 2023-05-09 19:02:24 +01:00
Dan Upton 972998203e
controller: deduplicate items in queue (#17168) 2023-05-09 18:14:20 +01:00
Dan Upton 6e1bc57469
Controller Runtime 2023-05-09 15:25:55 +01:00
Matt Keeler 34915670f2
Register new catalog & mesh protobuf types with the resource registry (#17225) 2023-05-08 15:36:35 -04:00
Semir Patel 991a002fcc
resource: List resources by owner (#17190) 2023-05-08 12:26:19 -05:00
Semir Patel 9fef1c7f17
Create tombstone on resource `Delete` (#17108) 2023-04-28 10:49:08 -05:00
Dan Upton eeaa636164
Cleanup from unblocking the pipeline 🧹 (#17121) 2023-04-26 13:59:58 +01:00
Semir Patel e7bb8fdf15
Fix or disable pipeline breaking changes that made it into main in last day or so (#17130)
* Fix straggler from renaming Register->RegisterTypes

* somehow a lint failure got through previously

* Fix lint-consul-retry errors

* adding in fix for success jobs getting skipped. (#17132)

* Temporarily disable inmem backend conformance test to get green pipeline

* Another test needs disabling

---------

Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-04-25 15:17:48 -05:00
Dan Upton b9c485dcb8
Controller Supervision (#17016) 2023-04-25 12:52:35 +01:00
Dan Upton ba4a314772
storage: fix bug where WatchList would (rarely) return duplicate events (#17067) 2023-04-25 11:48:13 +01:00
hashicorp-copywrite[bot] 9f81fc01e9
[COMPLIANCE] Add Copyright and License Headers (#16854)
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
2023-04-20 12:40:22 +00:00
Semir Patel 2f7d591702
Tenancy wildcard validaton for `Write`, `Read`, and `Delete` endpoints (#17004) 2023-04-17 16:33:20 -05:00
Semir Patel b8c9e133be
Add mutate hook to `Write` endpoint (#16958) 2023-04-12 16:50:07 -05:00
Semir Patel 3b83c7ee9a
Enforce ACLs on resource `Write` and `Delete` endpoints (#16956) 2023-04-12 16:22:44 -05:00
Dan Upton d595e6ade9
resource: `WriteStatus` endpoint (#16886) 2023-04-11 19:23:14 +01:00
Semir Patel 317240fca7
Resource validation hook for `Write` endpoint (#16950) 2023-04-11 06:55:32 -05:00
Semir Patel 686f49346c
Check acls on resource `Read`, `List`, and `WatchList` (#16842) 2023-04-11 06:10:14 -05:00
Dan Upton 4fa2537b3b
Resource `Write` endpoint (#16786) 2023-04-06 10:40:04 +01:00
Dan Upton 671d5825ca
Raft storage backend (#16619) 2023-04-04 17:30:06 +01:00
Dan Upton 651549c97d
storage: fix resource leak in Watch (#16817) 2023-03-31 13:24:19 +01:00
Ronald b64674623e
Copyright headers for missing files/folders (#16708)
* copyright headers for agent folder
2023-03-28 18:48:58 -04:00
Dan Upton 81df781e5f
Add storage backend interface and in-memory implementation (#16538)
Introduces `storage.Backend`, which will serve as the interface between the
Resource Service and the underlying storage system (Raft today, but in the
future, who knows!).

The primary design goal of this interface is to keep its surface area small,
and push as much functionality as possible into the layers above, so that new
implementations can be added with little effort, and easily proven to be
correct. To that end, we also provide a suite of "conformance" tests that can
be run against a backend implementation to check it behaves correctly.

In this commit, we introduce an initial in-memory storage backend, which is
suitable for tests and when running Consul in development mode. This backend is
a thin wrapper around the `Store` type, which implements a resource database
using go-memdb and our internal pub/sub system. `Store` will also be used to
handle reads in our Raft backend, and in the future, used as a local cache for
external storage systems.
2023-03-27 10:30:53 +01:00
Poonam Jadhav 9c64731a56
feat: add category annotation to RPC and gRPC methods (#16646) 2023-03-20 11:24:29 -04:00
Semir Patel 5a3fec6238
Basic resource type registry (#16622) 2023-03-14 13:30:25 -05:00
Matt Keeler 84156afe87
Remove private prefix from proto-gen-rpc-glue e2e test (#16433) 2023-03-03 14:05:14 -05:00
Matt Keeler 085c0addc0
Protobuf Refactoring for Multi-Module Cleanliness (#16302)
Protobuf Refactoring for Multi-Module Cleanliness

This commit includes the following:

Moves all packages that were within proto/ to proto/private
Rewrites imports to account for the packages being moved
Adds in buf.work.yaml to enable buf workspaces
Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml
Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes)

Why:

In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage.
There were some recent changes to have our own ratelimiting annotations.
The two combined were not working when I was trying to use them together (attempting to rebase another branch)
Buf workspaces should be the solution to the problem
Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root.
This resulted in proto file name conflicts in the Go global protobuf type registry.
The solution to that was to add in a private/ directory into the path within the proto/ directory.
That then required rewriting all the imports.

Is this safe?

AFAICT yes
The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc)
Other than imports, there were no changes to any generated code as a result of this.
2023-02-17 16:14:46 -05:00