Commit Graph

885 Commits

Author SHA1 Message Date
Melisa Griffin 606f8fbbab
Adds check to verify that the API Gateway is being created with at least one listener 2023-03-20 12:37:30 -04:00
Eric Haberkorn 7477f52a16
add sameness groups to discovery chains (#16671) 2023-03-20 09:12:37 -04:00
Eric Haberkorn eaa39f4ef5
add sameness group support to service resolver failover and redirects (#16664) 2023-03-17 10:48:06 -04:00
Eric Haberkorn 57e034b746
fix confusing spiffe ids in golden tests (#16643) 2023-03-15 14:30:36 -04:00
wangxinyi7 152c75349e
net 2731 ip config entry OSS version (#16642)
* ip config entry

* name changing

* move to ent

* ent version

* renaming

* change format

* renaming

* refactor

* add default values
2023-03-15 11:21:24 -07:00
John Maguire ff5887a99e
Update e2e tests for namespaces (#16627)
* Refactored "NewGatewayService" to handle namespaces, fixed
TestHTTPRouteFlattening test

* Fixed existing http_route tests for namespacing

* Squash aclEnterpriseMeta for ResourceRefs and HTTPServices, accept
namespace for creating connect services and regular services

* Use require instead of assert after creating namespaces in
http_route_tests

* Refactor NewConnectService and NewGatewayService functions to use cfg
objects to reduce number of method args

* Rename field on SidecarConfig in tests from `SidecarServiceName` to
`Name` to avoid stutter
2023-03-15 17:51:36 +00:00
Freddy 724b752ca7
Backport ENT-4704 (#16612) 2023-03-14 14:55:11 -06:00
Derek Menteer f2902e6608
Add sameness-group configuration entry. (#16608)
This commit adds a sameness-group config entry to the API and structs packages. It includes some validation logic and a new memdb index that tracks the default sameness-group for each partition. Sameness groups will simplify the effort of managing failovers / intentions / exports for peers and partitions.

Note that this change purely to introduce the configuration entry and does not include the full functionality of sameness-groups.
2023-03-13 16:19:11 -05:00
Eric Haberkorn e298f506a5
Add Peer Locality to Discovery Chains (#16588)
Add peer locality to discovery chains
2023-03-10 12:59:47 -05:00
Eric Haberkorn 57e2493415
allow setting locality on services and nodes (#16581) 2023-03-10 09:36:15 -05:00
Eric Haberkorn 89de91b263
fix bug that can lead to peering service deletes impacting the state of local services (#16570) 2023-03-08 11:24:03 -05:00
Eric Haberkorn dbaf8bf49c
add agent locality and replicate it across peer streams (#16522) 2023-03-07 14:05:23 -05:00
Melisa Griffin 129eca8fdb
NET-2903 Normalize weight for http routes (#16512)
* NET-2903 Normalize weight for http routes

* Update website/content/docs/connect/gateways/api-gateway/configuration/http-route.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2023-03-03 16:39:59 -05:00
Eric Haberkorn 5f81662066
Add support for failover policies (#16505) 2023-03-03 11:12:38 -05:00
Andrew Stucki 4b661d1e0c
Add ServiceResolver RequestTimeout for route timeouts to make TerminatingGateway upstream timeouts configurable (#16495)
* Leverage ServiceResolver ConnectTimeout for route timeouts to make TerminatingGateway upstream timeouts configurable

* Regenerate golden files

* Add RequestTimeout field

* Add changelog entry
2023-03-03 09:37:12 -05:00
Andrew Stucki 641737f32b
[API Gateway] Fix infinite loop in controller and binding non-accepted routes and gateways (#16377) 2023-02-22 14:55:40 -05:00
Andrew Stucki 0972697661
[API Gateway] Various fixes for Config Entry fields (#16347)
* [API Gateway] Various fixes for Config Entry fields

* simplify logic per PR review
2023-02-22 04:02:04 +00:00
Andrew Stucki 18e2ee77ca
[API Gateway] Fix targeting service splitters in HTTPRoutes (#16350)
* [API Gateway] Fix targeting service splitters in HTTPRoutes

* Fix test description
2023-02-22 03:48:26 +00:00
Derek Menteer ad865f549b
Fix issue with peer services incorrectly appearing as connect-enabled. (#16339)
Prior to this commit, all peer services were transmitted as connect-enabled
as long as a one or more mesh-gateways were healthy. With this change, there
is now a difference between typical services and connect services transmitted
via peering.

A service will be reported as "connect-enabled" as long as any of these
conditions are met:

1. a connect-proxy sidecar is registered for the service name.
2. a connect-native instance of the service is registered.
3. a service resolver / splitter / router is registered for the service name.
4. a terminating gateway has registered the service.
2023-02-21 13:59:36 -06:00
Andrew Stucki 7f9ec78932
[API Gateway] Validate listener name is not empty (#16340)
* [API Gateway] Validate listener name is not empty

* Update docstrings and test
2023-02-21 14:12:19 -05:00
Andrew Stucki 4607b535be
Fix HTTPRoute and TCPRoute expectation for enterprise metadata (#16322) 2023-02-17 17:28:49 -05:00
Andrew Stucki 15d2684ecc
Normalize all API Gateway references (#16316) 2023-02-17 21:37:34 +00:00
Matt Keeler 085c0addc0
Protobuf Refactoring for Multi-Module Cleanliness (#16302)
Protobuf Refactoring for Multi-Module Cleanliness

This commit includes the following:

Moves all packages that were within proto/ to proto/private
Rewrites imports to account for the packages being moved
Adds in buf.work.yaml to enable buf workspaces
Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml
Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes)

Why:

In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage.
There were some recent changes to have our own ratelimiting annotations.
The two combined were not working when I was trying to use them together (attempting to rebase another branch)
Buf workspaces should be the solution to the problem
Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root.
This resulted in proto file name conflicts in the Go global protobuf type registry.
The solution to that was to add in a private/ directory into the path within the proto/ directory.
That then required rewriting all the imports.

Is this safe?

AFAICT yes
The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc)
Other than imports, there were no changes to any generated code as a result of this.
2023-02-17 16:14:46 -05:00
Andrew Stucki 58801cc8aa
Add stricter validation and some normalization code for API Gateway ConfigEntries (#16304)
* Add stricter validation and some normalization code for API Gateway ConfigEntries
2023-02-17 19:22:01 +00:00
Andrew Stucki e4a992c581
Fix hostname alignment checks for HTTPRoutes (#16300)
* Fix hostname alignment checks for HTTPRoutes
2023-02-17 18:18:11 +00:00
Andrew Stucki b3ddd4d24e
Inline API Gateway TLS cert code (#16295)
* Include secret type when building resources from config snapshot

* First pass at generating envoy secrets from api-gateway snapshot

* Update comments for xDS update order

* Add secret type + corresponding golden files to existing tests

* Initialize test helpers for testing api-gateway resource generation

* Generate golden files for new api-gateway xDS resource test

* Support ADS for TLS certificates on api-gateway

* Configure TLS on api-gateway listeners

* Inline TLS cert code

* update tests

* Add SNI support so we can have multiple certificates

* Remove commented out section from helper

* regen deep-copy

* Add tcp tls test

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-02-17 12:46:03 -05:00
Derek Menteer 30112288c8
Fix mesh gateways incorrectly matching peer locality. (#16257)
Fix mesh gateways incorrectly matching peer locality.

This fixes an issue where local mesh gateways use an
incorrect address when attempting to forward traffic to a
peered datacenter. Prior to this change it would use the
lan address instead of the wan if the locality matched. This
should never be done for peering, since we must route all
traffic through the remote mesh gateway.
2023-02-16 09:22:41 -06:00
Nathan Coleman 514fb25a6f
Fix infinite recursion in inline-certificate config entry (#16276)
* Fix infinite recursion on InlineCertificateConfigEntry

GetNamespace() + GetMeta() were calling themselves. This change also simplifies by removing nil-checking to match pre-existing config entries

Co-Authored-By: Andrew Stucki <3577250+andrewstucki@users.noreply.github.com>

* Add tests for inline-certificate

* Add alias for private key field on inline-certificate

* Use valid certificate + private key for inline-certificate tests

---------

Co-authored-by: Andrew Stucki <3577250+andrewstucki@users.noreply.github.com>
2023-02-15 13:49:34 -06:00
Andrew Stucki 318ba215ab
[API Gateway] Add integration test for conflicted TCP listeners (#16225) 2023-02-10 11:34:01 -06:00
Thomas Eckert e81a0c2855
API Gateway to Ingress Gateway Snapshot Translation and Routes to Virtual Routers and Splitters (#16127)
* Stub proxycfg handler for API gateway

* Add Service Kind constants/handling for API Gateway

* Begin stubbing for SDS

* Add new Secret type to xDS order of operations

* Continue stubbing of SDS

* Iterate on proxycfg handler for API gateway

* Handle BoundAPIGateway config entry subscription in proxycfg-glue

* Add API gateway to config snapshot validation

* Add API gateway to config snapshot clone, leaf, etc.

* Subscribe to bound route + cert config entries on bound-api-gateway

* Track routes + certs on API gateway config snapshot

* Generate DeepCopy() for types used in watch.Map

* Watch all active references on api-gateway, unwatch inactive

* Track loading of initial bound-api-gateway config entry

* Use proper proto package for SDS mapping

* Use ResourceReference instead of ServiceName, collect resources

* Fix typo, add + remove TODOs

* Watch discovery chains for TCPRoute

* Add TODO for updating gateway services for api-gateway

* make proto

* Regenerate deep-copy for proxycfg

* Set datacenter on upstream ID from query source

* Watch discovery chains for http-route service backends

* Add ServiceName getter to HTTP+TCP Service structs

* Clean up unwatched discovery chains on API Gateway

* Implement watch for ingress leaf certificate

* Collect upstreams on http-route + tcp-route updates

* Remove unused GatewayServices update handler

* Remove unnecessary gateway services logic for API Gateway

* Remove outdate TODO

* Use .ToIngress where appropriate, including TODO for cleaning up

* Cancel before returning error

* Remove GatewayServices subscription

* Add godoc for handlerAPIGateway functions

* Update terminology from Connect => Consul Service Mesh

Consistent with terminology changes in https://github.com/hashicorp/consul/pull/12690

* Add missing TODO

* Remove duplicate switch case

* Rerun deep-copy generator

* Use correct property on config snapshot

* Remove unnecessary leaf cert watch

* Clean up based on code review feedback

* Note handler properties that are initialized but set elsewhere

* Add TODO for moving helper func into structs pkg

* Update generated DeepCopy code

* gofmt

* Begin stubbing for SDS

* Start adding tests

* Remove second BoundAPIGateway case in glue

* TO BE PICKED: fix formatting of str

* WIP

* Fix merge conflict

* Implement HTTP Route to Discovery Chain config entries

* Stub out function to create discovery chain

* Add discovery chain merging code (#16131)

* Test adding TCP and HTTP routes

* Add some tests for the synthesizer

* Run go mod tidy

* Pairing with N8

* Run deep copy

* Clean up GatewayChainSynthesizer

* Fix missing assignment of BoundAPIGateway topic

* Separate out synthesizeChains and toIngressTLS

* Fix build errors

* Ensure synthesizer skips non-matching routes by protocol

* Rebase on N8s work

* Generate DeepCopy() for API gateway listener types

* Improve variable name

* Regenerate DeepCopy() code

* Fix linting issue

* fix protobuf import

* Fix more merge conflict errors

* Fix synthesize test

* Run deep copy

* Add URLRewrite to proto

* Update agent/consul/discoverychain/gateway_tcproute.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Remove APIGatewayConfigEntry that was extra

* Error out if route kind is unknown

* Fix formatting errors in proto

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-02-09 17:58:55 +00:00
Andrew Stucki f4210d47dd
Add basic smoke test to make sure an APIGateway runs (#16217) 2023-02-09 11:32:10 -05:00
Andrew Stucki 0891b4554d
Clean-up Gateway Controller Binding Logic (#16214)
* Fix detecting when a route doesn't bind to a gateway because it's already bound

* Clean up status setting code

* rework binding a bit

* More cleanup

* Flatten all files

* Fix up docstrings
2023-02-09 10:17:25 -05:00
Nathan Coleman 72a73661c9
Implement APIGateway proxycfg snapshot (#16194)
* Stub proxycfg handler for API gateway

* Add Service Kind constants/handling for API Gateway

* Begin stubbing for SDS

* Add new Secret type to xDS order of operations

* Continue stubbing of SDS

* Iterate on proxycfg handler for API gateway

* Handle BoundAPIGateway config entry subscription in proxycfg-glue

* Add API gateway to config snapshot validation

* Add API gateway to config snapshot clone, leaf, etc.

* Subscribe to bound route + cert config entries on bound-api-gateway

* Track routes + certs on API gateway config snapshot

* Generate DeepCopy() for types used in watch.Map

* Watch all active references on api-gateway, unwatch inactive

* Track loading of initial bound-api-gateway config entry

* Use proper proto package for SDS mapping

* Use ResourceReference instead of ServiceName, collect resources

* Fix typo, add + remove TODOs

* Watch discovery chains for TCPRoute

* Add TODO for updating gateway services for api-gateway

* make proto

* Regenerate deep-copy for proxycfg

* Set datacenter on upstream ID from query source

* Watch discovery chains for http-route service backends

* Add ServiceName getter to HTTP+TCP Service structs

* Clean up unwatched discovery chains on API Gateway

* Implement watch for ingress leaf certificate

* Collect upstreams on http-route + tcp-route updates

* Remove unused GatewayServices update handler

* Remove unnecessary gateway services logic for API Gateway

* Remove outdate TODO

* Use .ToIngress where appropriate, including TODO for cleaning up

* Cancel before returning error

* Remove GatewayServices subscription

* Add godoc for handlerAPIGateway functions

* Update terminology from Connect => Consul Service Mesh

Consistent with terminology changes in https://github.com/hashicorp/consul/pull/12690

* Add missing TODO

* Remove duplicate switch case

* Rerun deep-copy generator

* Use correct property on config snapshot

* Remove unnecessary leaf cert watch

* Clean up based on code review feedback

* Note handler properties that are initialized but set elsewhere

* Add TODO for moving helper func into structs pkg

* Update generated DeepCopy code

* gofmt

* Generate DeepCopy() for API gateway listener types

* Improve variable name

* Regenerate DeepCopy() code

* Fix linting issue

* Temporarily remove the secret type from resource generation
2023-02-08 15:52:12 -06:00
Kyle Havlovitz 898e59b13c
Add the `operator usage instances` command and api endpoint (#16205)
This endpoint shows total services, connect service instances and
billable service instances in the local datacenter or globally. Billable
instances = total service instances - connect services - consul server instances.
2023-02-08 12:07:21 -08:00
Andrew Stucki df03b45bbc
Add additional controller implementations (#16188)
* Add additional controller implementations

* remove additional interface

* Fix comparison checks and mark unused contexts

* Switch to time.Now().UTC()

* Add a pointer helper for shadowing loop variables

* Extract anonymous functions for readability

* clean up logging

* Add Type to the Condition proto

* Update some comments and add additional space for readability

* Address PR feedback

* Fix up dirty checks and change to pointer receiver
2023-02-08 14:50:17 -05:00
Paul Banks 5397e9ee7f
Adding experimental support for a more efficient LogStore implementation (#16176)
* Adding experimental support for a more efficient LogStore implementation

* Adding changelog entry

* Fix go mod tidy issues
2023-02-08 16:50:22 +00:00
skpratt 9199e99e21
Update token language to distinguish Accessor and Secret ID usage (#16044)
* remove legacy tokens

* remove lingering legacy token references from docs

* update language and naming for token secrets and accessor IDs

* updates all tokenID references to clarify accessorID

* remove token type references and lookup tokens by accessorID index

* remove unnecessary constants

* replace additional tokenID param names

* Add warning info for deprecated -id parameter

Co-authored-by: Paul Glass <pglass@hashicorp.com>

* Update field comment

Co-authored-by: Paul Glass <pglass@hashicorp.com>

---------

Co-authored-by: Paul Glass <pglass@hashicorp.com>
2023-02-07 12:26:30 -06:00
skpratt a010902978
Remove legacy acl policies (#15922)
* remove legacy tokens

* remove legacy acl policies

* flatten test policies to *_prefix

* address oss feedback re: phrasing and tests
2023-02-06 15:35:52 +00:00
Derek Menteer 2f149d60cc
[OSS] Add Peer field to service-defaults upstream overrides (#15956)
* Add Peer field to service-defaults upstream overrides.

* add api changes, compat mode for service default overrides

* Fixes based on testing

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-02-03 10:51:53 -05:00
Andrew Stucki 1fbfb5905b
APIGateway HTTPRoute scaffolding (#15859)
* Stub Config Entries for Consul Native API Gateway (#15644)

* Add empty InlineCertificate struct and protobuf

* apigateway stubs

* new files

* Stub HTTPRoute in api pkg

* checkpoint

* Stub HTTPRoute in structs pkg

* Simplify api.APIGatewayConfigEntry to be consistent w/ other entries

* Update makeConfigEntry switch, add docstring for HTTPRouteConfigEntry

* Add TCPRoute to MakeConfigEntry, return unique Kind

* proto generated files

* Stub BoundAPIGatewayConfigEntry in agent

Since this type is only written by a controller and read by xDS, it doesn't need to be defined in the `api` pkg

* Add RaftIndex to APIGatewayConfigEntry stub

* Add new config entry kinds to validation allow-list

* Add RaftIndex to other added config entry stubs

* fix panic

* Update usage metrics assertions to include new cfg entries

* Regenerate proto w/ Go 1.19

* Run buf formatter on config_entry.proto

* Add Meta and acl.EnterpriseMeta to all new ConfigEntry types

* Remove optional interface method Warnings() for now

Will restore later if we wind up needing it

* Remove unnecessary Services field from added config entry types

* Implement GetMeta(), GetEnterpriseMeta() for added config entry types

* Add meta field to proto, name consistently w/ existing config entries

* Format config_entry.proto

* Add initial implementation of CanRead + CanWrite for new config entry types

* Add unit tests for decoding of new config entry types

* Add unit tests for parsing of new config entry types

* Add unit tests for API Gateway config entry ACLs

* Return typed PermissionDeniedError on BoundAPIGateway CanWrite

* Add unit tests for added config entry ACLs

* Add BoundAPIGateway type to AllConfigEntryKinds

* Return proper kind from BoundAPIGateway

* Add docstrings for new config entry types

* Add missing config entry kinds to proto def

* Update usagemetrics_oss_test.go

* Use utility func for returning PermissionDeniedError

* Add BoundAPIGateway to proto def

Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Add APIGateway validation

* Fix comment

* Add additional validations

* Add cert ref validation

* Add protobuf definitions

* Tabs to spaces

* Fix up field types

* Add API structs

* Move struct fields around a bit

* EventPublisher subscriptions for Consul Native API Gateway (#15757)

* Create new event topics in subscribe proto
* Add tests for PBSubscribe func
* Make configs singular, add all configs to PBToStreamSubscribeRequest
* Add snapshot methods
* Add config_entry_events tests
* Add config entry kind to topic for new configs
* Add unit tests for snapshot methods
* Start adding integration test
* Test using the new controller code
* Update agent/consul/state/config_entry_events.go
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
* Check value of error
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Add controller stubs for API Gateway (#15837)

* update initial stub implementation

* move files, clean up mutex references

* Remove embed, use idiomatic names for constructors

* Remove stray file introduced in merge

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Initial server-side and proto defs

* drop trailing whitespace

* Add APIGateway validation (#15847)

* Add APIGateway validation

* Fix comment

* Add additional validations

* Add cert ref validation

* Add protobuf definitions

* Tabs to spaces

* Fix up field types

* Add API structs

* Move struct fields around a bit

* APIGateway InlineCertificate validation (#15856)

* Add APIGateway validation

* Add additional validations

* Add protobuf definitions

* Tabs to spaces

* Add API structs

* Move struct fields around a bit

* Add validation for InlineCertificate

* Fix ACL test

* APIGateway BoundAPIGateway validation (#15858)

* Add APIGateway validation

* Fix comment

* Add additional validations

* Add cert ref validation

* Add protobuf definitions

* Tabs to spaces

* Fix up field types

* Add API structs

* Move struct fields around a bit

* Add validation for BoundAPIGateway

* drop trailing whitespace

* APIGateway TCPRoute validation (#15855)

* Add APIGateway validation

* Fix comment

* Add additional validations

* Add cert ref validation

* Add protobuf definitions

* Tabs to spaces

* Fix up field types

* Add API structs

* Move struct fields around a bit

* Add TCPRoute normalization and validation

* Address PR feedback

* Add forgotten Status

* Add some more field docs in api package

* Fix test

* Fix bad merge

* Remove duplicate helpers

* Fix up proto defs

* Fix up stray changes

* remove extra newline

---------

Co-authored-by: Thomas Eckert <teckert@hashicorp.com>
Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>
2023-02-01 07:59:49 -05:00
Derek Menteer 1b02749375
Add extension validation on config save and refactor extensions. (#16110) 2023-01-30 15:35:26 -06:00
Andrew Stucki da99514ac8
Add a server-only method for updating ConfigEntry Statuses (#16053)
* Add a server-only method for updating ConfigEntry Statuses

* Address PR feedback

* Regen proto
2023-01-27 14:34:11 -05:00
skpratt ad43846755
Remove legacy acl tokens (#15947)
* remove legacy tokens

* Update test comment

Co-authored-by: Paul Glass <pglass@hashicorp.com>

* fix imports

* update docs for additional CLI changes

* add test case for anonymous token

* set deprecated api fields to json ignore and fix patch errors

* update changelog to breaking-change

* fix import

* update api docs to remove legacy reference

* fix docs nav data

---------

Co-authored-by: Paul Glass <pglass@hashicorp.com>
2023-01-27 09:17:07 -06:00
Thomas Eckert 7814471159
Match route and listener protocols when binding (#16057)
* Add GatewayMeta for matching routes to listeners based on protocols
* Add GetGatewayMeta
* Apply suggestions from code review
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
* Make GatewayMeta private
* Bound -> BoundGateway
* Document gatewayMeta more
* Simplify conditional
* Parallelize tests and simplify bind conditional
* gofmt
* 💧 getGatewayMeta
---------
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-01-27 09:41:03 -05:00
Thomas Eckert 20146f2916
Implement BindRoutesToGateways (#15950)
* Stub out bind code
* Move into a new package and flesh out binding
* Fill in the actual binding logic
* Bind to all listeners if not specified
* Move bind code up to gateways package
* Fix resource type check
* Add UpsertRoute to listeners
* Add RemoveRoute to listener
* Implement binding as associated functions
* Pass in gateways to BindRouteToGateways
* Add a bunch of tests
* Fix hopping from one listener on a gateway to another
* Remove parents from HTTPRoute
* Apply suggestions from code review
* Fix merge conflict
* Unify binding into a single variadic function 🙌 @nathancoleman
* Remove vestigial error
* Add TODO on protocol check
2023-01-20 15:11:16 -05:00
Thomas Eckert 13da1a5285
Native API Gateway Config Entries (#15897)
* Stub Config Entries for Consul Native API Gateway (#15644)
* Add empty InlineCertificate struct and protobuf
* apigateway stubs
* Stub HTTPRoute in api pkg
* Stub HTTPRoute in structs pkg
* Simplify api.APIGatewayConfigEntry to be consistent w/ other entries
* Update makeConfigEntry switch, add docstring for HTTPRouteConfigEntry
* Add TCPRoute to MakeConfigEntry, return unique Kind
* Stub BoundAPIGatewayConfigEntry in agent
* Add RaftIndex to APIGatewayConfigEntry stub
* Add new config entry kinds to validation allow-list
* Add RaftIndex to other added config entry stubs
* Update usage metrics assertions to include new cfg entries
* Add Meta and acl.EnterpriseMeta to all new ConfigEntry types
* Remove unnecessary Services field from added config entry types
* Implement GetMeta(), GetEnterpriseMeta() for added config entry types
* Add meta field to proto, name consistently w/ existing config entries
* Format config_entry.proto
* Add initial implementation of CanRead + CanWrite for new config entry types
* Add unit tests for decoding of new config entry types
* Add unit tests for parsing of new config entry types
* Add unit tests for API Gateway config entry ACLs
* Return typed PermissionDeniedError on BoundAPIGateway CanWrite
* Add unit tests for added config entry ACLs
* Add BoundAPIGateway type to AllConfigEntryKinds
* Return proper kind from BoundAPIGateway
* Add docstrings for new config entry types
* Add missing config entry kinds to proto def
* Update usagemetrics_oss_test.go
* Use utility func for returning PermissionDeniedError
* EventPublisher subscriptions for Consul Native API Gateway (#15757)
* Create new event topics in subscribe proto
* Add tests for PBSubscribe func
* Make configs singular, add all configs to PBToStreamSubscribeRequest
* Add snapshot methods
* Add config_entry_events tests
* Add config entry kind to topic for new configs
* Add unit tests for snapshot methods
* Start adding integration test
* Test using the new controller code
* Update agent/consul/state/config_entry_events.go
* Check value of error
* Add controller stubs for API Gateway (#15837)
* update initial stub implementation
* move files, clean up mutex references
* Remove embed, use idiomatic names for constructors
* Remove stray file introduced in merge
* Add APIGateway validation (#15847)
* Add APIGateway validation
* Add additional validations
* Add cert ref validation
* Add protobuf definitions
* Fix up field types
* Add API structs
* Move struct fields around a bit
* APIGateway InlineCertificate validation (#15856)
* Add APIGateway validation
* Add additional validations
* Add protobuf definitions
* Tabs to spaces
* Add API structs
* Move struct fields around a bit
* Add validation for InlineCertificate
* Fix ACL test
* APIGateway BoundAPIGateway validation (#15858)
* Add APIGateway validation
* Add additional validations
* Add cert ref validation
* Add protobuf definitions
* Fix up field types
* Add API structs
* Move struct fields around a bit
* Add validation for BoundAPIGateway
* APIGateway TCPRoute validation (#15855)
* Add APIGateway validation
* Add additional validations
* Add cert ref validation
* Add protobuf definitions
* Fix up field types
* Add API structs
* Add TCPRoute normalization and validation
* Add forgotten Status
* Add some more field docs in api package
* Fix test
* Format imports
* Rename snapshot test variable names
* Add plumbing for Native API GW Subscriptions (#16003)

Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>
Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-01-18 22:14:34 +00:00
Matt Keeler 5afd4657ec
Protobuf Modernization (#15949)
* Protobuf Modernization

Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf

Marshallers (protobuf and json) needed some changes to account for different APIs.

Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package.

This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces.

* Fix go-mod-tidy make target to work on all modules
2023-01-11 09:39:10 -05:00
Chris S. Kim a7b34d50fc
Output user-friendly name for anonymous token (#15884) 2023-01-09 12:28:53 -06:00
Eric Haberkorn 8d923c1789
Add the Lua Envoy extension (#15906) 2023-01-06 12:13:40 -05:00
Florian Apolloner 077b0a48a3
Allow Operator Generated bootstrap token (#14437)
Add support to provide an initial token via the bootstrap HTTP API, similar to hashicorp/nomad#12520
2023-01-04 20:19:33 +00:00
Dan Stough b3bd3a6586
[OSS] feat: access logs for listeners and listener filters (#15864)
* feat: access logs for listeners and listener filters

* changelog

* fix integration test
2022-12-22 15:18:15 -05:00
Nitya Dhanushkodi 24f01f96b1
add extensions for local service to GetExtensionConfigurations (#15871)
This gets the extensions information for the local service into the snapshot and ExtensionConfigurations for a proxy. It grabs the extensions from config entries and puts them in structs.NodeService.Proxy field, which already is copied into the config snapshot.

Also:
* add EnvoyExtensions to api.AgentService so that it matches structs.NodeService
2022-12-22 10:03:33 -08:00
Nitya Dhanushkodi c7ef04c597
[OSS] extensions: refactor PluginConfiguration into a more generic type ExtensionConfiguration (#15846)
* extensions: refactor PluginConfiguration into a more generic type
ExtensionConfiguration

Also:
* adds endpoints configuration to lambda golden tests
* uses string constant for builtin/aws/lambda
Co-authored-by: Eric <eric@haberkorn.co>
2022-12-20 22:26:20 -08:00
Nitya Dhanushkodi d382ca0aec
extensions: refactor serverless plugin to use extensions from config entry fields (#15817)
docs: update config entry docs and the Lambda manual registration docs

Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>
Co-authored-by: Eric <eric@haberkorn.co>
2022-12-19 12:19:37 -08:00
Dan Stough 233dbcb67f
feat: add access logging API to proxy defaults (#15780) 2022-12-13 14:52:18 -05:00
cskh 04bf24c8c1
feat(ingress-gateway): support outlier detection of upstream service for ingress gateway (#15614)
* feat(ingress-gateway): support outlier detection of upstream service for ingress gateway

* changelog

Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>
2022-12-13 11:51:37 -05:00
Derek Menteer e87d35e313
Fix DialedDirectly configuration for Consul dataplane. (#15760)
Fix DialedDirectly configuration for Consul dataplane.
2022-12-13 09:16:31 -06:00
James Oulman 7e78fb7818
Add support for configuring Envoys route idle_timeout (#14340)
* Add idleTimeout

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
2022-11-29 17:43:15 -05:00
Freddy 706866fa00
Ensure that NodeDump imported nodes are filtered (#15356) 2022-11-14 12:35:20 -07:00
Freddy c58f86a00f
Fixup authz for data imported from peers (#15347)
There are a few changes that needed to be made to to handle authorizing
reads for imported data:

- If the data was imported from a peer we should not attempt to read the
  data using the traditional authz rules. This is because the name of
  services/nodes in a peer cluster are not equivalent to those of the
  importing cluster.

- If the data was imported from a peer we need to check whether the
  token corresponds to a service, meaning that it has service:write
  permissions, or to a local read only token that can read all
  nodes/services in a namespace.

This required changes at the policyAuthorizer level, since that is the
only view available to OSS Consul, and at the enterprise
partition/namespace level.
2022-11-14 11:36:27 -07:00
malizz b51f0e25e9
update ACLs for cluster peering (#15317)
* update ACLs for cluster peering

* add changelog

* Update .changelog/15317.txt

Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>

Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>
2022-11-09 13:02:58 -08:00
Derek Menteer 418bd62c44
Fix mesh gateway configuration with proxy-defaults (#15186)
* Fix mesh gateway proxy-defaults not affecting upstreams.

* Clarify distinction with upstream settings

Top-level mesh gateway mode in proxy-defaults and service-defaults gets
merged into NodeService.Proxy.MeshGateway, and only gets merged with
the mode attached to an an upstream in proxycfg/xds.

* Fix mgw mode usage for peered upstreams

There were a couple issues with how mgw mode was being handled for
peered upstreams.

For starters, mesh gateway mode from proxy-defaults
and the top-level of service-defaults gets stored in
NodeService.Proxy.MeshGateway, but the upstream watch for peered data
was only considering the mesh gateway config attached in
NodeService.Proxy.Upstreams[i]. This means that applying a mesh gateway
mode via global proxy-defaults or service-defaults on the downstream
would not have an effect.

Separately, transparent proxy watches for peered upstreams didn't
consider mesh gateway mode at all.

This commit addresses the first issue by ensuring that we overlay the
upstream config for peered upstreams as we do for non-peered. The second
issue is addressed by re-using setupWatchesForPeeredUpstream when
handling transparent proxy updates.

Note that for transparent proxies we do not yet support mesh gateway
mode per upstream, so the NodeService.Proxy.MeshGateway mode is used.

* Fix upstream mesh gateway mode handling in xds

This commit ensures that when determining the mesh gateway mode for
peered upstreams we consider the NodeService.Proxy.MeshGateway config as
a baseline.

In absense of this change, setting a mesh gateway mode via
proxy-defaults or the top-level of service-defaults will not have an
effect for peered upstreams.

* Merge service/proxy defaults in cfg resolver

Previously the mesh gateway mode for connect proxies would be
merged at three points:

1. On servers, in ComputeResolvedServiceConfig.
2. On clients, in MergeServiceConfig.
3. On clients, in proxycfg/xds.

The first merge returns a ServiceConfigResponse where there is a
top-level MeshGateway config from proxy/service-defaults, along with
per-upstream config.

The second merge combines per-upstream config specified at the service
instance with per-upstream config specified centrally.

The third merge combines the NodeService.Proxy.MeshGateway
config containing proxy/service-defaults data with the per-upstream
mode. This third merge is easy to miss, which led to peered upstreams
not considering the mesh gateway mode from proxy-defaults.

This commit removes the third merge, and ensures that all mesh gateway
config is available at the upstream. This way proxycfg/xds do not need
to do additional overlays.

* Ensure that proxy-defaults is considered in wc

Upstream defaults become a synthetic Upstream definition under a
wildcard key "*". Now that proxycfg/xds expect Upstream definitions to
have the final MeshGateway values, this commit ensures that values from
proxy-defaults/service-defaults are the default for this synthetic
upstream.

* Add changelog.

Co-authored-by: freddygv <freddy@hashicorp.com>
2022-11-09 10:14:29 -06:00
Derek Menteer b64972d486
Bring back parameter ServerExternalAddresses in GenerateToken endpoint (#15267)
Re-add ServerExternalAddresses parameter in GenerateToken endpoint

This reverts commit 5e156772f6
and adds extra functionality to support newer peering behaviors.
2022-11-08 14:55:18 -06:00
Chris S. Kim bde57c0dd0 Regenerate files according to 1.19.2 formatter 2022-10-24 16:12:08 -04:00
Chris S. Kim 29a297d3e9
Refactor client RPC timeouts (#14965)
Fix an issue where rpc_hold_timeout was being used as the timeout for non-blocking queries. Users should be able to tune read timeouts without fiddling with rpc_hold_timeout. A new configuration `rpc_read_timeout` is created.

Refactor some implementation from the original PR 11500 to remove the misleading linkage between RPCInfo's timeout (used to retry in case of certain modes of failures) and the client RPC timeouts.
2022-10-18 15:05:09 -04:00
Chris S. Kim 3d2dffff16
Merge pull request #13388 from deblasis/feature/health-checks_windows_service
Feature: Health checks windows service
2022-10-17 09:26:19 -04:00
Dan Upton e6b55d1d81
perf: remove expensive reflection from xDS hot path (#14934)
Replaces the reflection-based implementation of proxycfg's
ConfigSnapshot.Clone with code generated by deep-copy.

While load testing server-based xDS (for consul-dataplane) we discovered
this method is extremely expensive. The ConfigSnapshot struct, directly
or indirectly, contains a copy of many of the structs in the agent/structs
package, which creates a large graph for copystructure.Copy to traverse
at runtime, on every proxy reconfiguration.
2022-10-14 10:26:42 +01:00
Derek Menteer caa1396255 Add remote peer partition and datacenter info. 2022-10-13 10:37:41 -05:00
Eric Haberkorn 1633cf20ea
Make the mesh gateway changes to allow `local` mode for cluster peering data plane traffic (#14817)
Make the mesh gateway changes to allow `local` mode for cluster peering data plane traffic
2022-10-06 09:54:14 -04:00
Alex Oskotsky 13da2c5fad
Add the ability to retry on reset connection to service-routers (#12890) 2022-10-05 13:06:44 -04:00
Eric Haberkorn 1b565444be
Rename `PeerName` to `Peer` on prepared queries and exported services (#14854) 2022-10-04 14:46:15 -04:00
Freddy d9fe3578ac
Merge pull request #14734 from hashicorp/NET-643-update-mesh-gateway-envoy-config-for-inbound-peering-control-plane-traffic 2022-10-03 12:54:11 -06:00
freddygv b15d41534f Update xds generation for peering over mesh gws
This commit adds the xDS resources needed for INBOUND traffic from peer
clusters:

- 1 filter chain for all inbound peering requests.
- 1 cluster for all inbound peering requests.
- 1 endpoint per voting server with the gRPC TLS port configured.

There is one filter chain and cluster because unlike with WAN
federation, peer clusters will not attempt to dial individual servers.
Peer clusters will only dial the local mesh gateway addresses.
2022-10-03 12:42:27 -06:00
cskh 69f40df548
feat(ingress gateway: support configuring limits in ingress-gateway c… (#14749)
* feat(ingress gateway: support configuring limits in ingress-gateway config entry

- a new Defaults field with max_connections, max_pending_connections, max_requests
  is added to ingress gateway config entry
- new field max_connections, max_pending_connections, max_requests in
  individual services to overwrite the value in Default
- added unit test and integration test
- updated doc

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
2022-09-28 14:56:46 -04:00
malizz 84b0f408fa
Support Stale Queries for Trust Bundle Lookups (#14724)
* initial commit

* add tags, add conversations

* add test for query options utility functions

* update previous tests

* fix test

* don't error out on empty context

* add changelog

* update decode config
2022-09-28 09:56:59 -07:00
Derek Menteer aa4709ab74
Add envoy connection balancing. (#14616)
Add envoy connection balancing config.
2022-09-26 11:29:06 -05:00
Alessandro De Blasis 461b42ed48 fix(check): added missing OSService props 2022-09-21 13:10:21 +01:00
freddygv 02d3ce1039 Add server certificate manager
This certificate manager will request a leaf certificate for server
agents and then keep them up to date.
2022-09-16 17:57:10 -06:00
freddygv 0e5131bd33 Generate ACL token for server management
This commit introduces a new ACL token used for internal server
management purposes.

It has a few key properties:
- It has unlimited permissions.
- It is persisted through Raft as System Metadata rather than in the
ACL tokens table. This is to avoid users seeing or modifying it.
- It is re-generated on leadership establishment.
2022-09-16 17:54:34 -06:00
Kyle Havlovitz 60cee76746
Merge pull request #14516 from hashicorp/ca-ttl-fixes
Fix inconsistent TTL behavior in CA providers
2022-09-13 16:07:36 -07:00
Kyle Havlovitz d67bccd210 Update intermediate pki mount/role when reconfiguring Vault provider 2022-09-13 15:42:26 -07:00
Kyle Havlovitz f46955101a connect/ca: Clarify behavior around IntermediateCertTTL in CA config 2022-09-13 15:42:26 -07:00
DanStough 0150e88200 feat: add PeerThroughMeshGateways to mesh config 2022-09-13 17:19:54 -04:00
cskh f22685b969
Config-entry: Support proxy config in service-defaults (#14395)
* Config-entry: Support proxy config in service-defaults

* Update website/content/docs/connect/config-entries/service-defaults.mdx

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
2022-09-12 10:41:58 -04:00
skpratt 3bf1edfb3f
move port and default check logic to locked step (#14057) 2022-09-06 19:35:31 -05:00
Freddy f4dfd42e0a
Add SpiffeID for Consul server agents (#14485)
Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>

By adding a SpiffeID for server agents, servers can now request a leaf
certificate from the Connect CA.

This new Spiffe ID has a key property: servers are identified by their
datacenter name and trust domain. All servers that share these
attributes will share a ServerURI.

The aim is to use these certificates to verify the server name of ANY
server in a Consul datacenter.
2022-09-06 17:58:13 -06:00
malizz f1054dada9
fix TestProxyConfigEntry (#14435) 2022-09-01 11:37:47 -07:00
malizz b3ac8f48ca
Add additional parameters to envoy passive health check config (#14238)
* draft commit

* add changelog, update test

* remove extra param

* fix test

* update type to account for nil value

* add test for custom passive health check

* update comments and tests

* update description in docs

* fix missing commas
2022-09-01 09:59:11 -07:00
malizz a80e0bcd00
validate args before deleting proxy defaults (#14290)
* validate args before deleting proxy defaults

* add changelog

* validate name when normalizing proxy defaults

* add test for proxyConfigEntry

* add comments
2022-08-31 13:03:38 -07:00
Alessandro De Blasis 60c7c831c6 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service 2022-08-30 18:49:20 +01:00
Eric Haberkorn 3726a0ab7a
Finish up cluster peering failover (#14396) 2022-08-30 11:46:34 -04:00
Eric Haberkorn 1099665473
Update the structs and discovery chain for service resolver redirects to cluster peers. (#14366) 2022-08-29 09:51:32 -04:00
Alessandro De Blasis f3437eaf05 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service
Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2022-08-28 18:09:31 +01:00
Eric Haberkorn 58901ad7df
Cluster peering failover disco chain changes (#14296) 2022-08-23 09:13:43 -04:00
cskh 527ebd068a
fix: missing MaxInboundConnections field in service-defaults config entry (#14072)
* fix:  missing max_inbound_connections field in merge config
2022-08-19 14:11:21 -04:00
James Hartig f92883bbce Use the maximum jitter when calculating the timeout
The timeout should include the maximum possible
jitter since the server will randomly add to it's
timeout a jitter. If the server's timeout is less
than the client's timeout then the client will
return an i/o deadline reached error.

Before:
```
time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644'
rpc error making call: i/o deadline reached
real    10m11.469s
user    0m0.018s
sys     0m0.023s
```

After:
```
time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644'
[...]
real    10m35.835s
user    0m0.021s
sys     0m0.021s
```
2022-08-17 10:24:09 -04:00
Eric Haberkorn 1a73b0ca20
Add `Targets` field to service resolver failovers. (#14162)
This field will be used for cluster peering failover.
2022-08-15 09:20:25 -04:00
Alessandro De Blasis 5dee555888 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service
Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2022-08-15 08:26:55 +01:00
Alessandro De Blasis ab611eabc3 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service
Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2022-08-15 08:09:56 +01:00
Eric Haberkorn ebd5513d4b
Refactor failover code to use Envoy's aggregate clusters (#14178) 2022-08-12 14:30:46 -04:00