Commit Graph

5303 Commits

Author SHA1 Message Date
Chris S. Kim ecdcde4309
CE commit (#18583) 2023-08-25 12:47:20 -04:00
John Murret 051f250edb
NET-5338 - NET-5338 - Run a v2 mode xds server (#18579)
* NET-5338 - NET-5338 - Run a v2 mode xds server

* fix linting
2023-08-24 16:44:14 -06:00
Semir Patel 2225bf0550
resource: Make resource writestatus tenancy aware (#18577) 2023-08-24 19:18:47 +00:00
John Maguire 59ab57f350
NET-5147: Added placeholder structs for JWT functionality (#18575)
* Added placeholder structs for JWT functionality

* Added watches for CE vs ENT

* Add license header

* Undo plumbing work

* Add context arg
2023-08-24 15:07:14 -04:00
Semir Patel 067a0112e2
resource: Make resource listbyowner tenancy aware (#18566) 2023-08-24 10:49:46 -05:00
Chris S. Kim 82993fcc4f
CE port of enterprise extension (#18572)
CE commit
2023-08-24 15:43:26 +00:00
cskh b37587bb2c
bug: prevent go routine leakage due to existing DeferCheck (#18558)
* bug: prevent go routine leakage due to existing DeferCheck

* add changelog
2023-08-23 10:33:07 -04:00
R.B. Boyer 8a931241f2
chore: fix missing/incorrect license headers (#18555) 2023-08-22 17:23:54 -05:00
Ashwin Venkatesh 4f9955d91e
Update trust bundle into proxy-state-template (#18550) 2023-08-22 19:38:31 +00:00
Semir Patel 53e28a4963
OSS -> CE (community edition) changes (#18517) 2023-08-22 09:46:03 -05:00
Semir Patel 6d22179625
resource: Make resource watchlist tenancy aware (#18539) 2023-08-21 15:02:23 -05:00
John Murret 217d305b38
NET-4943 - Implement ProxyTracker (#18535) 2023-08-21 14:08:13 -04:00
John Murret 9ea182f6ad
NET-4858 - xds v2 - implement base connect proxy functionality for routes (#18501)
* NET-4853 - xds v2 - implement base connect proxy functionality for clusters

* NET-4853 - xds v2 - implement base connect proxy functionality for clusters

* NET-4932 - xds v2 - implement base connect proxy functionality for endpoints

* Update endpoints_test.go

* gofmt

* NET-4858 - Make connect proxy route tests pass using xds v2

* Update endpoints_test.go

* Update naming.go

* use alsoRunTestForV2

* remove unused makeAddress

* gofmt

* fixing clusters
2023-08-17 21:04:53 +00:00
John Murret 92cfb4a07e
NET-4932 - xds v2 - implement base connect proxy functionality for endpoints (#18500)
* NET-4853 - xds v2 - implement base connect proxy functionality for clusters

* NET-4853 - xds v2 - implement base connect proxy functionality for clusters

* NET-4932 - xds v2 - implement base connect proxy functionality for endpoints

* Update endpoints_test.go

* gofmt

* Update naming.go
2023-08-17 19:55:54 +00:00
John Murret b80c5258fa
NET-4853 - xds v2 - implement base connect proxy functionality for clusters (#18499) 2023-08-17 14:43:21 -04:00
Semir Patel e6c1c479b7
resource: Make resource delete tenancy aware (#18476)
resource: Make resource delete tenancy awarae
2023-08-16 11:44:10 -05:00
Semir Patel 217107f627
resource: Make resource list tenancy aware (#18475) 2023-08-15 16:57:59 -05:00
Nitya Dhanushkodi 6b7ccd06cf
[NET-4799] [OSS] xdsv2: listeners L4 support for connect proxies (#18436)
* refactor to avoid future import cycles
2023-08-15 11:57:07 -07:00
hashicorp-copywrite[bot] 5fb9df1640
[COMPLIANCE] License changes (#18443)
* Adding explicit MPL license for sub-package

This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.

* Adding explicit MPL license for sub-package

This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.

* Updating the license from MPL to Business Source License

Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at <Blog URL>, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.

* add missing license headers

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

---------

Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
2023-08-11 09:12:13 -04:00
John Maguire df11e4e7b4
APIGW: Update HTTPRouteConfigEntry for JWT Auth (#18422)
* Updated httproute config entry for JWT Filters

* Added manual deepcopy method for httproute jwt filter

* Fix test

* Update JWTFilter to be in oss file

* Add changelog

* Add build tags for deepcopy oss file
2023-08-10 21:23:42 +00:00
John Maguire 6c8ca0f89d
NET-4984: Update APIGW Config Entries for JWT Auth (#18366)
* Added oss config entries for Policy and JWT on APIGW

* Updated structs for config entry

* Updated comments, ran deep-copy

* Move JWT configuration into OSS file

* Add in the config entry OSS file for jwts

* Added changelog

* fixing proto spacing

* Moved to using manually written deep copy method

* Use pointers for override/default fields in apigw config entries

* Run gen scripts for changed types
2023-08-10 19:49:51 +00:00
Michael Zalimeni 05604eeec1
[NET-5217] [OSS] Derive sidecar proxy locality from parent service (#18437)
* Add logging to locality policy application

In OSS, this is currently a no-op.

* Inherit locality when registering sidecars

When sidecar locality is not explicitly configured, inherit locality
from the proxied service.
2023-08-10 14:00:44 -04:00
Semir Patel bee12c6b1f
resource: Make resource write tenancy aware (#18423) 2023-08-10 09:53:38 -05:00
wangxinyi7 facd5b0ec1
fix the error in ent repo (#18421)
fix the error in ent repo
2023-08-09 09:36:58 -07:00
sarahalsmiller e235c8be3c
NET-5115 Add retry + timeout filters for api-gateway (#18324)
* squash, implement retry/timeout in consul core

* update tests
2023-08-08 16:39:46 -05:00
cskh 43d8898e08
bump testcontainers-go from 0.22.0 and remove pinned go version in in… (#18395)
* bump testcontainers-go from 0.22.0 and remove pinned go version in integ test

* go mod tidy

* Replace deprecated target.Authority with target.URL.Host
2023-08-08 18:08:14 +00:00
Semir Patel 63cc037110
resource: Make resource read tenancy aware (#18397) 2023-08-07 16:37:03 -05:00
Ashesh Vidyut 417ae9fc39
Fix #17730 - Dev mode has new line (#18367)
* adding new line only in case of pretty in url not in dev mode

* change log added
2023-08-05 08:15:24 +05:30
wangxinyi7 1f28ac2664
expose grpc as http endpoint (#18221)
expose resource grpc endpoints as http endpoints
2023-08-04 11:27:48 -07:00
Jeremy Jacobson 8e5e16de60
Fix policy lookup to allow for slashes (#18347)
* Fix policy lookup to allow for slashes

* Fix suggestions

* Fix other test

* Revert some lines
2023-08-03 13:21:43 -07:00
Dan Stough 284e3bdb54
[OSS] test: xds coverage for routes (#18369)
test: xds coverage for routes
2023-08-03 15:03:02 -04:00
Ashvitha 828567c62e
[HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration (#18168)
* OTElExporter now uses an EndpointProvider to discover the endpoint

* OTELSink uses a ConfigProvider to obtain filters and labels configuration

* improve tests for otel_sink

* Regex logic is moved into client for a method on the TelemetryConfig object

* Create a telemetry_config_provider and update deps to use it

* Fix conversion

* fix import newline

* Add logger to hcp client and move telemetry_config out of the client.go file

* Add a telemetry_config.go to refactor client.go

* Update deps

* update hcp deps test

* Modify telemetry_config_providers

* Check for nil filters

* PR review updates

* Fix comments and move around pieces

* Fix comments

* Remove context from client struct

* Moved ctx out of sink struct and fixed filters, added a test

* Remove named imports, use errors.New if not fformatting

* Remove HCP dependencies in telemetry package

* Add success metric and move lock only to grab the t.cfgHahs

* Update hash

* fix nits

* Create an equals method and add tests

* Improve telemetry_config_provider.go tests

* Add race test

* Add missing godoc

* Remove mock for MetricsClient

* Avoid goroutine test panics

* trying to kick CI lint issues by upgrading mod

* imprve test code and add hasher for testing

* Use structure logging for filters, fix error constants, and default to allow all regex

* removed hashin and modify logic to simplify

* Improve race test and fix PR feedback by removing hash equals and avoid testing the timer.Ticker logic, and instead unit test

* Ran make go-mod-tidy

* Use errtypes in the test

* Add changelog

* add safety check for exporter endpoint

* remove require.Contains by using error types, fix structure logging, and fix success metric typo in exporter

* Fixed race test to have changing config values

* Send success metric before modifying config

* Avoid the defer and move the success metric under
2023-08-01 17:20:18 -04:00
Jeremy Jacobson 6424ef6a56
[CC-5719] Add support for builtin global-read-only policy (#18319)
* [CC-5719] Add support for builtin global-read-only policy

* Add changelog

* Add read-only to docs

* Fix some minor issues.

* Change from ReplaceAll to Sprintf

* Change IsValidPolicy name to return an error instead of bool

* Fix PolicyList test

* Fix other tests

* Apply suggestions from code review

Co-authored-by: Paul Glass <pglass@hashicorp.com>

* Fix state store test for policy list.

* Fix naming issues

* Update acl/validation.go

Co-authored-by: Chris Thain <32781396+cthain@users.noreply.github.com>

* Update agent/consul/acl_endpoint.go

---------

Co-authored-by: Paul Glass <pglass@hashicorp.com>
Co-authored-by: Chris Thain <32781396+cthain@users.noreply.github.com>
2023-08-01 17:12:14 +00:00
Michael Zalimeni b1b05f0bac
[NET-4703] Prevent partial application of Envoy extensions (#18068)
Prevent partial application of Envoy extensions

Ensure that non-required extensions do not change xDS resources before
exiting on failure by cloning proto messages prior to applying each
extension.

To support this change, also move `CanApply` checks up a layer and make
them prior to attempting extension application, s.t. we avoid
unnecessary copies where extensions can't be applied.

Last, ensure that we do not allow panics from `CanApply` or `Extend`
checks to escape the attempted extension application.
2023-07-31 15:24:33 -04:00
cui fliter 18a5edd232
docs: Fix some comments (#17118)
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-07-31 10:56:09 -07:00
Ronald 356b29bf35
Stop JWT provider from being written in non default namespace (#18325) 2023-07-31 09:13:16 -04:00
Florian Apolloner 6ada2e05ff
Fix topology view when displaying mixed connect-native/normal services. (#13023)
* Fix topoloy intention with mixed connect-native/normal services.

If a service is registered twice, once with connect-native and once
without, the topology views would prune the existing intentions. This
change brings the code more in line with the transparent proxy behavior.

* Dedupe nodes in the ServiceTopology ui endpoint (like done with tags).

* Consider a service connect-native as soon as one instance is.
2023-07-31 08:10:55 -04:00
Nathan Coleman 5caa0ae3f5
api-gateway: subscribe to bound-api-gateway only after receiving api-gateway (#18291)
* api-gateway: subscribe to bound-api-gateway only after receiving api-gateway

This fixes a race condition due to our dependency on having the listener(s) from the api-gateway config entry in order to fully and properly process the resources on the bound-api-gateway config entry.

* Apply suggestions from code review

* Add changelog entry
2023-07-26 16:02:04 -04:00
cskh 31d2813714
member cli: add -filter expression to flags (#18223)
* member cli: add -filter expression to flags

* changelog

* update doc

* Add test cases

* use quote
2023-07-25 13:54:52 -04:00
Dan Stough 8e3a1ddeb6
[OSS] Improve xDS Code Coverage - Endpoints and Misc (#18222)
test: improve xDS endpoints code coverage
2023-07-21 17:48:25 -04:00
Jeremy Jacobson 6671d7ebd7
[CC-5718] Remove HCP token requirement during bootstrap (#18140)
* [CC-5718] Remove HCP token requirement during bootstrap

* Re-add error for loading HCP management token

* Remove old comment

* Add changelog entry

* Remove extra validation line

* Apply suggestions from code review

Co-authored-by: lornasong <lornasong@users.noreply.github.com>

---------

Co-authored-by: lornasong <lornasong@users.noreply.github.com>
2023-07-21 10:33:22 -07:00
Dan Stough 2793761702
[OSS] Improve xDS Code Coverage - Clusters (#18165)
test: improve xDS cluster code coverage
2023-07-20 18:02:21 -04:00
cskh 5cd287660a
docs: fix the description of client rpc (#18206) 2023-07-20 16:34:36 -04:00
Blake Covarrubias 2c5a09bb0a
Explicitly enable WebSocket upgrades (#18150)
This PR explicitly enables WebSocket upgrades in Envoy's UpgradeConfig for all
proxy types. (API Gateway, Ingress, and Sidecar.)

Fixes #8283
2023-07-20 13:24:43 -07:00
Semir Patel ada767fc9f
resource: Pass resource to Write ACL hook instead of just resource Id [NET-4908] (#18192) 2023-07-20 12:06:29 -05:00
Ronald 18bc04165c
Improve XDS test coverage: JWT auth edition (#18183)
* Improve XDS test coverage: JWT auth edition

more tests

* test: xds coverage for jwt listeners

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-07-19 17:19:00 -04:00
Semir Patel 003370ded0
Call resource mutate hook before validate hook (NET-4907) (#18178) 2023-07-19 13:10:57 -05:00
Dan Stough 33d898b857
[OSS] test: improve xDS listener code coverage (#18138)
test: improve xDS listener code coverage
2023-07-17 13:49:40 -04:00
Ronald bcc6a9d752
Use JWT-auth filter in metadata mode & Delegate validation to RBAC filter (#18062)
### Description

<!-- Please describe why you're making this change, in plain English.
-->

- Currently the jwt-auth filter doesn't take into account the service
identity when validating jwt-auth, it only takes into account the path
and jwt provider during validation. This causes issues when multiple
source intentions restrict access to an endpoint with different JWT
providers.
- To fix these issues, rather than use the JWT auth filter for
validation, we use it in metadata mode and allow it to forward the
successful validated JWT token payload to the RBAC filter which will
make the decisions.

This PR ensures requests with and without JWT tokens successfully go
through the jwt-authn filter. The filter however only forwards the data
for successful/valid tokens. On the RBAC filter level, we check the
payload for claims and token issuer + existing rbac rules.

### Testing & Reproduction steps

<!--

* In the case of bugs, describe how to replicate
* If any manual tests were done, document the steps and the conditions
to replicate
* Call out any important/ relevant unit tests, e2e tests or integration
tests you have added or are adding

-->

- This test covers a multi level jwt requirements (requirements at top
level and permissions level). It also assumes you have envoy running,
you have a redis and a sidecar proxy service registered, and have a way
to generate jwks with jwt. I mostly use:
https://www.scottbrady91.com/tools/jwt for this.

- first write your proxy defaults
```
Kind = "proxy-defaults"
name = "global"
config {
  protocol = "http"
}
```
- Create two providers 
```
Kind = "jwt-provider"
Name = "auth0"
Issuer = "https://ronald.local"

JSONWebKeySet = {
    Local = {
     JWKS = "eyJrZXlzIjog....."
    }
}
```

```
Kind = "jwt-provider"
Name = "okta"
Issuer = "https://ronald.local"

JSONWebKeySet = {
   Local = {
     JWKS = "eyJrZXlzIjogW3...."
    }
}
```

- add a service intention
```
Kind = "service-intentions"
Name = "redis"

JWT = {
  Providers = [
    {
      Name = "okta"
    },
  ]
}

Sources = [
  {
    Name = "*"
    Permissions = [{
      Action = "allow"
      HTTP = {
        PathPrefix = "/workspace"
      }
      JWT = {
        Providers = [
          {
            Name = "okta"
            VerifyClaims = [
              {
                  Path = ["aud"]
                  Value = "my_client_app"
              },
              {
                Path = ["sub"]
                Value = "5be86359073c434bad2da3932222dabe"
              }
            ]
          },
        ]
      }

    },
    {
      Action = "allow"
      HTTP = {
        PathPrefix = "/"
      }
      JWT = {
        Providers = [
          {
            Name = "auth0"
          },
        ]
      }

    }]
  }
]
```
- generate 3 jwt tokens: 1 from auth0 jwks, 1 from okta jwks with
different claims than `/workspace` expects and 1 with correct claims
- connect to your envoy (change service and address as needed) to view
logs and potential errors. You can add: `-- --log-level debug` to see
what data is being forwarded
```
consul connect envoy -sidecar-for redis1 -grpc-addr 127.0.0.1:8502
```
- Make the following requests: 
```
curl -s -H "Authorization: Bearer $Auth0_TOKEN" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v

RBAC filter denied

curl -s -H "Authorization: Bearer $Okta_TOKEN_with_wrong_claims" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v

RBAC filter denied

curl -s -H "Authorization: Bearer $Okta_TOKEN_with_correct_claims" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v

Successful request
```


### TODO

* [x] Update test coverage
* [ ] update integration tests (follow-up PR)
* [x] appropriate backport labels added
2023-07-17 11:32:49 -04:00
Poonam Jadhav 5930518489
fix: update delegateMock used in ENT (#18149)
### Description

<!-- Please describe why you're making this change, in plain English.
-->
The mock is used in `http_ent_test` file which caused lint failures. For
OSS->ENT parity adding the same change here.

### Links

<!--

Include any links here that might be helpful for people reviewing your
PR (Tickets, GH issues, API docs, external benchmarks, tools docs, etc).
If there are none, feel free to delete this section.

Please be mindful not to leak any customer or confidential information.
HashiCorp employees may want to use our internal URL shortener to
obfuscate links.

-->

Identified in OSS->ENT [merge
PR](https://github.com/hashicorp/consul-enterprise/pull/6328)

### PR Checklist

* [ ] ~updated test coverage~
* [ ] ~external facing docs updated~
* [x] appropriate backport labels added
* [ ] ~not a security concern~
2023-07-17 09:44:49 -04:00
wangxinyi7 e7194787a7
re org resource type registry (#18133) 2023-07-14 18:00:17 -07:00
John Murret 691bc9673a
add a conditional around setting LANFilter.AllSegments to make sure it is valid (#18139)
### Description

This is to correct a code problem because this assumes all segments, but
when you get to Enterprise, you can be in partition that is not the
default partition, in which case specifying all segments does not
validate and fails. This is to correct the setting of this filter with
`AllSegments` to `true` to only occur when in the the `default`
partition.

### Testing & Reproduction steps

<!--

* In the case of bugs, describe how to replicate
* If any manual tests were done, document the steps and the conditions
to replicate
* Call out any important/ relevant unit tests, e2e tests or integration
tests you have added or are adding

-->

### Links

<!--

Include any links here that might be helpful for people reviewing your
PR (Tickets, GH issues, API docs, external benchmarks, tools docs, etc).
If there are none, feel free to delete this section.

Please be mindful not to leak any customer or confidential information.
HashiCorp employees may want to use our internal URL shortener to
obfuscate links.

-->

### PR Checklist

* [ ] updated test coverage
* [ ] external facing docs updated
* [ ] appropriate backport labels added
* [ ] not a security concern
2023-07-14 14:53:44 -06:00
Chris S. Kim 747a4c73c1
Fix bug with Vault CA provider (#18112)
Updating RootPKIPath but not IntermediatePKIPath would not update 
leaf signing certs with the new root. Unsure if this happens in practice 
but manual testing showed it is a bug that would break mesh and agent 
connections once the old root is pruned.
2023-07-14 15:58:33 -04:00
Poonam Jadhav 5208ea90e4
NET-4657/add resource service client (#18053)
### Description

<!-- Please describe why you're making this change, in plain English.
-->
Dan had already started on this
[task](https://github.com/hashicorp/consul/pull/17849) which is needed
to start building the HTTP APIs. This just needed some cleanup to get it
ready for review.

Overview:

- Rename `internalResourceServiceClient` to
`insecureResourceServiceClient` for name consistency
- Configure a `secureResourceServiceClient` with auth enabled

### PR Checklist

* [ ] ~updated test coverage~
* [ ] ~external facing docs updated~
* [x] appropriate backport labels added
* [ ] ~not a security concern~
2023-07-14 14:09:02 -04:00
Vijay 2f20c77e4d
Displays Consul version of each nodes in UI nodes section (#17754)
* update UINodes and UINodeInfo response with consul-version info added as NodeMeta, fetched from serf members

* update test cases TestUINodes, TestUINodeInfo

* added nil check for map

* add consul-version in local agent node metadata

* get consul version from serf member and add this as node meta in catalog register request

* updated ui mock response to include consul versions as node meta

* updated ui trans and added version as query param to node list route

* updates in ui templates to display consul version with filter and sorts

* updates in ui - model class, serializers,comparators,predicates for consul version feature

* added change log for Consul Version Feature

* updated to get version from consul service, if for some reason not available from serf

* updated changelog text

* updated dependent testcases

* multiselection version filter

* Update agent/consul/state/catalog.go

comments updated

Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>

---------

Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
2023-07-12 13:34:39 -06:00
Tom Davies f472164f05
Pass configured role name to Vault for AWS auth in Connect CA (#17885) 2023-07-12 08:24:12 -07:00
Dan Stough da79997f3d
test: fix FIPS inline cert test message (#18076) 2023-07-11 11:28:27 -04:00
Dan Stough 1b08626358
[OSS] Fix initial_fetch_timeout to wait for all xDS resources (#18024)
* fix(connect): set initial_fetch_time to wait indefinitely

* changelog

* PR feedback 1
2023-07-10 17:08:06 -04:00
Fulvio f4b08040fd
Add verify server hostname to tls default (#17155) 2023-07-10 10:34:41 -05:00
Ronald ada3938115
Add first integration test for jwt auth with intention (#18005) 2023-07-06 07:27:30 -04:00
Poonam Jadhav 8af4ad178c
feat: include nodes count in operator usage endpoint and cli command (#17939)
* feat: update operator usage api endpoint to include nodes count

* feat: update operator usange cli command to includes nodes count
2023-07-05 11:23:29 -04:00
Derek Menteer 0094dbf312
Fix incorrect protocol for transparent proxy upstreams. (#17894)
This PR fixes a bug that was introduced in:
https://github.com/hashicorp/consul/pull/16021

A user setting a protocol in proxy-defaults would cause tproxy implicit
upstreams to not honor the upstream service's protocol set in its
`ServiceDefaults.Protocol` field, and would instead always use the
proxy-defaults value.

Due to the fact that upstreams configured with "tcp" can successfully contact
upstream "http" services, this issue was not recognized until recently (a
proxy-defaults with "tcp" and a listening service with "http" would make
successful requests, but not the opposite).

As a temporary work-around, users experiencing this issue can explicitly set
the protocol on the `ServiceDefaults.UpstreamConfig.Overrides`, which should
take precedence.

The fix in this PR removes the proxy-defaults protocol from the wildcard
upstream that tproxy uses to configure implicit upstreams. When the protocol
was included, it would always overwrite the value during discovery chain
compilation, which was not correct. The discovery chain compiler also consumes
proxy defaults to determine the protocol, so simply excluding it from the
wildcard upstream config map resolves the issue.
2023-07-05 09:32:10 -05:00
Ronald 80394278b8
Expose JWKS cluster config through JWTProviderConfigEntry (#17978)
* Expose JWKS cluster config through JWTProviderConfigEntry

* fix typos, rename trustedCa to trustedCA
2023-07-04 09:12:06 -04:00
Chris Thain 0b1299c28d
Remove duplicate and unused newDecodeConfigEntry func (#17979) 2023-06-30 09:39:54 -07:00
Chris S. Kim 50a9d1b696
Remove POC code (#17974) 2023-06-30 14:05:13 +00:00
Ashesh Vidyut 2af6bc434a
feature - [NET - 4005] - [Supportability] Reloadable Configuration - enable_debug (#17565)
* # This is a combination of 9 commits.
# This is the 1st commit message:

init without tests

# This is the commit message #2:

change log

# This is the commit message #3:

fix tests

# This is the commit message #4:

fix tests

# This is the commit message #5:

added tests

# This is the commit message #6:

change log breaking change

# This is the commit message #7:

removed breaking change

# This is the commit message #8:

fix test

# This is the commit message #9:

keeping the test behaviour same

* # This is a combination of 12 commits.
# This is the 1st commit message:

init without tests

# This is the commit message #2:

change log

# This is the commit message #3:

fix tests

# This is the commit message #4:

fix tests

# This is the commit message #5:

added tests

# This is the commit message #6:

change log breaking change

# This is the commit message #7:

removed breaking change

# This is the commit message #8:

fix test

# This is the commit message #9:

keeping the test behaviour same

# This is the commit message #10:

made enable debug atomic bool

# This is the commit message #11:

fix lint

# This is the commit message #12:

fix test true enable debug

* parent 10f500e895d92cc3691ade7b74a33db755d22039
author absolutelightning <ashesh.vidyut@hashicorp.com> 1687352587 +0530
committer absolutelightning <ashesh.vidyut@hashicorp.com> 1687352592 +0530

init without tests

change log

fix tests

fix tests

added tests

change log breaking change

removed breaking change

fix test

keeping the test behaviour same

made enable debug atomic bool

fix lint

fix test true enable debug

using enable debug in agent as atomic bool

test fixes

fix tests

fix tests

added update on correct locaiton

fix tests

fix reloadable config enable debug

fix tests

fix init and acl 403

* revert commit
2023-06-30 08:30:29 +05:30
Ronald 1512ea307e
Dynamically create jwks clusters for jwt-providers (#17944) 2023-06-29 20:37:40 +00:00
Ranjandas 1b1f33f224
Fixes Secondary ConnectCA update (#17846)
This fixes a bug that was identified which resulted in subsequent
ConnectCA configuration update not to persist in the cluster.
2023-06-29 14:24:24 +00:00
John Maguire 67a239a821
Ensure RSA keys are at least 2048 bits in length (#17911)
* Ensure RSA keys are at least 2048 bits in length

* Add changelog

* update key length check for FIPS compliance

* Fix no new variables error and failing to return when error exists from
validating

* clean up code for better readability

* actually return value
2023-06-28 15:34:09 +00:00
Ronald 767ef2dd4c
Allow service identity tokens the ability to read jwt-providers (#17893)
* Allow service identity tokens the ability to read jwt-providers

* more tests

* service_prefix tests
2023-06-27 16:03:43 +00:00
Alex Simenduev 33a2d90852
Fix a bug that wrongly trims domains when there is an overlap with DC name (#17160)
* Fix a bug that wrongly trims domains when there is an overlap with DC name

Before this change, when DC name and domain/alt-domain overlap, the domain name incorrectly trimmed from the query.

Example:

Given: datacenter = dc-test, alt-domain = test.consul.
Querying for "test-node.node.dc-test.consul" will faile, because the
code was trimming "test.consul" instead of just ".consul"

This change, fixes the issue by adding dot (.) before trimming

* trimDomain: ensure domain trimmed without modyfing original domains

* update changelog

---------

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
2023-06-26 10:57:11 -04:00
Dan Upton b117eb0126
resource: enforce consistent naming of resource types (#17611)
For consistency, resource type names must follow these rules:

- `Group` must be snake case, and in most cases a single word.
- `GroupVersion` must be lowercase, start with a "v" and end with a number.
- `Kind` must be pascal case.

These were chosen because they map to our protobuf type naming
conventions.
2023-06-26 13:25:14 +01:00
cskh f16c5d87ab
watch: support -filter for consul watch: checks, services, nodes, service (#17780)
* watch: support -filter for watch checks

* Add filter for watch nodes, services, and service
- unit test added
- Add changelog
- update doc
2023-06-23 12:00:46 -04:00
Chris Thain 366bd6f89f
ext-authz Envoy extension: support `localhost` as a valid target URI. (#17821) 2023-06-21 13:42:42 -07:00
Chris S. Kim a4653de8da
CA provider doc updates and Vault provider minor update (#17831)
Update CA provider docs

Clarify that providers can differ between
primary and secondary datacenters

Provide a comparison chart for consul vs
vault CA providers

Loosen Vault CA provider validation for RootPKIPath

Update Vault CA provider documentation
2023-06-21 19:34:42 +00:00
George Bolo 82441a27fa
fixes #17732 - AccessorID in request body should be optional when updating ACL token (#17739)
* AccessorID in request body should be optional when updating ACL token

* add a test case

* fix test case

* add changelog entry for PR #17739
2023-06-21 13:31:40 -05:00
Eric Haberkorn a3ba559149
Make locality aware routing xDS changes (#17826) 2023-06-21 12:39:53 -04:00
Paul Glass d2363eb711
Test permissive mTLS filter chain not configured with tproxy disabled (#17747) 2023-06-20 09:49:50 -05:00
chappie 5352ccf8ed
HCP Add node id/name to config (#17750) 2023-06-16 18:44:13 +00:00
Ronald 5f95f5f6d8
Stop referenced jwt providers from being deleted (#17755)
* Stop referenced jwt providers from being deleted
2023-06-16 10:31:53 -04:00
Michael Zalimeni 265c003033
Add Patch index to Prop Override validation errors (#17777)
When a patch is found invalid, include its index for easier debugging
when multiple patches are provided.
2023-06-16 09:37:47 -04:00
Michael Zalimeni f9aa7aebb3
Property Override validation improvements (#17759)
* Reject inbound Prop Override patch with Services

Services filtering is only supported for outbound TrafficDirection patches.

* Improve Prop Override unexpected type validation

- Guard against additional invalid parent and target types
- Add specific error handling for Any fields (unsupported)
2023-06-15 13:51:47 -04:00
Derek Menteer 04edace1de
Fix issue with streaming service health watches. (#17775)
Fix issue with streaming service health watches.

This commit fixes an issue where the health streams were unaware of service
export changes. Whenever an exported-services config entry is modified, it is
effectively an ACL change.

The bug would be triggered by the following situation:

- no services are exported
- an upstream watch to service X is spawned
- the streaming backend filters out data for service X (due to lack of exports)
- service X is finally exported

In the situation above, the streaming backend does not trigger a refresh of its
data.  This means that any events that were supposed to have been received prior
to the export are NOT backfilled, and the watches never see service X spawning.

We currently have decided to not trigger a stream refresh in this situation due
to the potential for a thundering herd effect (touching exports would cause a
re-fetch of all watches for that partition, potentially).  Therefore, a local
blocking-query approach was added by this commit for agentless.

It's also worth noting that the streaming subscription is currently bypassed
most of the time with agentful, because proxycfg has a `req.Source.Node != ""`
which prevents the `streamingEnabled` check from passing.  This means that while
agents should technically have this same issue, they don't experience it with
mesh health watches.

Note that this is a temporary fix that solves the issue for proxycfg, but not
service-discovery use cases.
2023-06-15 12:46:58 -05:00
Eric Haberkorn 0994ccf162
validate localities on agent configs and registration endpoints (#17712) 2023-06-15 10:01:04 -04:00
chappie 7ab287c1d5
Add truncation to body (#17723) 2023-06-14 11:17:13 -07:00
Chris Thain 9289e680d6
OSS merge: Update error handling login when applying extensions (#17740) 2023-06-14 10:04:40 -07:00
Ashesh Vidyut fa40654885
[NET-3865] [Supportability] Additional Information in the output of 'consul operator raft list-peers' (#17582)
* init

* fix tests

* added -detailed in docs

* added change log

* fix doc

* checking for entry in map

* fix tests

* removed detailed flag

* removed detailed flag

* revert unwanted changes

* removed unwanted changes

* updated change log

* pr review comment changes

* pr comment changes single API instead of two

* fix change log

* fix tests

* fix tests

* fix test operator raft endpoint test

* Update .changelog/17582.txt

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* nits

* updated docs

---------

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>
2023-06-14 15:12:50 +00:00
R.B. Boyer 72f991d8d3
agent: remove agent cache dependency from service mesh leaf certificate management (#17075)
* agent: remove agent cache dependency from service mesh leaf certificate management

This extracts the leaf cert management from within the agent cache.

This code was produced by the following process:

1. All tests in agent/cache, agent/cache-types, agent/auto-config,
   agent/consul/servercert were run at each stage.

    - The tests in agent matching .*Leaf were run at each stage.

    - The tests in agent/leafcert were run at each stage after they
      existed.

2. The former leaf cert Fetch implementation was extracted into a new
   package behind a "fake RPC" endpoint to make it look almost like all
   other cache type internals.

3. The old cache type was shimmed to use the fake RPC endpoint and
   generally cleaned up.

4. I selectively duplicated all of Get/Notify/NotifyCallback/Prepopulate
   from the agent/cache.Cache implementation over into the new package.
   This was renamed as leafcert.Manager.

    - Code that was irrelevant to the leaf cert type was deleted
      (inlining blocking=true, refresh=false)

5. Everything that used the leaf cert cache type (including proxycfg
   stuff) was shifted to use the leafcert.Manager instead.

6. agent/cache-types tests were moved and gently replumbed to execute
   as-is against a leafcert.Manager.

7. Inspired by some of the locking changes from derek's branch I split
   the fat lock into N+1 locks.

8. The waiter chan struct{} was eventually replaced with a
   singleflight.Group around cache updates, which was likely the biggest
   net structural change.

9. The awkward two layers or logic produced as a byproduct of marrying
   the agent cache management code with the leaf cert type code was
   slowly coalesced and flattened to remove confusion.

10. The .*Leaf tests from the agent package were copied and made to work
    directly against a leafcert.Manager to increase direct coverage.

I have done a best effort attempt to port the previous leaf-cert cache
type's tests over in spirit, as well as to take the e2e-ish tests in the
agent package with Leaf in the test name and copy those into the
agent/leafcert package to get more direct coverage, rather than coverage
tangled up in the agent logic.

There is no net-new test coverage, just coverage that was pushed around
from elsewhere.
2023-06-13 10:54:45 -05:00
Eric Haberkorn 0a1efe73f3
Refactor disco chain prioritize by locality structs (#17696)
This includes prioritize by localities on disco chain targets rather than
resolvers, allowing different targets within the same partition to have
different policies.
2023-06-13 11:03:30 -04:00
Dan Stough bba5cd8455
fix: stop peering delete routine on leader loss (#17483) 2023-06-13 10:20:56 -04:00
Chris Thain a8f1350835
ENT merge of ext-authz extension updates (#17684) 2023-06-13 06:57:11 -07:00
Chris Thain c04c122ef3
Default `ProxyType` for builtin extensions (#17657) 2023-06-12 10:47:31 -07:00
Nathan Coleman 1074252361
api-gateway: stop adding all header filters to virtual host when generating xDS (#17644)
* Add header filter to api-gateway xDS golden test

* Stop adding all header filters to virtual host when generating xDS for api-gateway

* Regenerate xDS golden file for api-gateway w/ header filter
2023-06-12 12:06:04 -04:00
Matt Keeler baaf6d84c7
Add generic experiments configuration and use it to enable catalog v2 resources (#17604)
* Add generic experiments configuration and use it to enable catalog v2 resources

* Run formatting with -s as CI will validate that this has been done
2023-06-12 11:32:43 -04:00
R.B. Boyer ec347ef01d
sort some imports that are wonky between oss and ent (#17637) 2023-06-09 11:30:56 -05:00
Andrew Stucki 3cb70566a9
[API Gateway] Fix rate limiting for API gateways (#17631)
* [API Gateway] Fix rate limiting for API gateways

* Add changelog

* Fix failing unit tests

* Fix operator usage tests for api package
2023-06-09 08:22:32 -04:00
Andrew Stucki 9a4f503b2b
[API Gateway] Fix trust domain for external peered services in synthesis code (#17609)
* [API Gateway] Fix trust domain for external peered services in synthesis code

* Add changelog
2023-06-08 12:18:17 -04:00
Eric Haberkorn 779647b948
Add Envoy and Consul version constraints to Envoy extensions (#17612) 2023-06-08 10:26:11 -04:00
Ronald 8118aae5c1
Add writeAuditRPCEvent to agent_oss (#17607)
* Add writeAuditRPCEvent to agent_oss

* fix the other diffs

* backport change log
2023-06-07 22:35:48 +00:00
Michael Zalimeni 1db02a0349
Disable terminating-gateway for property-override (#17605)
More validation is needed to ensure this behaves as expected; in the
meantime, align with docs and disable this proxy type.
2023-06-07 19:39:25 +00:00
R.B. Boyer 820cdf53da
fix some testing.T retry.R mixups (#17600)
Fix some linter warnings before updating the lint-consul-retry code in hashicorp/lint-consul-retry#4
2023-06-07 13:53:27 -05:00
Dhia Ayachi 39d4aaf224
fix rate limiting mapping to be the same between api and struct packages (#17599) 2023-06-07 14:50:22 -04:00
skpratt a35cafa728
update tests for fips (#17592) 2023-06-07 10:57:56 -05:00
Michael Zalimeni 2dd5551003
Fix Property Override Services parsing (#17584)
Ensure that the embedded api struct is properly parsed when
deserializing config containing a set ResourceFilter.Services field.

Also enhance existing integration test to guard against bugs and
exercise this field.
2023-06-06 15:40:37 -04:00
Andrew Stucki f9d9d4db60
Fix subscribing/fetching objects not in the default partition (#17581)
* Fix subscribing/fetching objects not in the default namespace

* add changelog
2023-06-06 09:09:33 -04:00
Matt Keeler 77f44fa878
Various bits of cleanup detected when using Go Workspaces (#17462)
TLDR with many modules the versions included in each diverged quite a bit. Attempting to use Go Workspaces produces a bunch of errors.

This commit:

1. Fixes envoy-library-references.sh to work again
2. Ensures we are pulling in go-control-plane@v0.11.0 everywhere (previously it was at that version in some modules and others were much older)
3. Remove one usage of golang/protobuf that caused us to have a direct dependency on it.
4. Remove deprecated usage of the Endpoint field in the grpc resolver.Target struct. The current version of grpc (v1.55.0) has removed that field and recommended replacement with URL.Opaque and calls to the Endpoint() func when needing to consume the previous field.
4. `go work init <all the paths to go.mod files>` && `go work sync`. This syncrhonized versions of dependencies from the main workspace/root module to all submodules
5. Updated .gitignore to ignore the go.work and go.work.sum files. This seems to be standard practice at the moment.
6. Update doc comments in protoc-gen-consul-rate-limit to be go fmt compatible
7. Upgraded makefile infra to perform linting, testing and go mod tidy on all modules in a flexible manner.
8. Updated linter rules to prevent usage of golang/protobuf
9. Updated a leader peering test to account for an extra colon in a grpc error message.
2023-06-05 16:08:39 -04:00
malizz 8617f8af16
continue anti-entropy sync when failures exist (#17560) 2023-06-05 12:16:21 -07:00
Andrew Stucki 4ddb88ec7e
Fix up case where subscription is terminated due to ACLs changing or a snapshot restore occurring (#17566)
* Fix up case where subscription is terminated due to ACLs changing or a snapshot restore occurring

* Add changelog entry

* Switch to use errors.Is
2023-06-05 13:10:17 -04:00
cskh cf4059f3ce
chore: fix the error message format (#17554) 2023-06-02 13:37:44 +00:00
Michael Zalimeni ad03a5d0f2
Avoid panic applying TProxy Envoy extensions (#17537)
When UpstreamEnvoyExtender was introduced, some code was left duplicated
between it and BasicEnvoyExtender. One path in that code panics when a
TProxy listener patch is attempted due to no upstream data in
RuntimeConfig matching the local service (which would only happen in
rare cases).

Instead, we can remove the special handling of upstream VIPs from
BasicEnvoyExtender entirely, greatly simplifying the listener filter
patch code and avoiding the panic. UpstreamEnvoyExtender, which needs
this code to function, is modified to ensure a panic does not occur.

This also fixes a second regression in which the Lua extension was not
applied to TProxy outbound listeners.
2023-06-01 13:04:39 -04:00
Andrew Stucki ca12ce926b
[API Gateway] Fix use of virtual resolvers in HTTPRoutes (#17055)
* [API Gateway] Fix use of virtual resolvers in routes

* Add changelog entry
2023-05-31 16:58:40 -04:00
Derek Menteer ba26e188d5
Fix tproxy failover issue with sameness groups (#17533)
Sameness groups with default-for-failover enabled did not function properly with
tproxy whenever all instances of the service disappeared from the local cluster.
This occured, because there were no corresponding resolvers (due to the implicit
failover policy) which caused VIPs to be deallocated.

This ticket expands upon the VIP allocations so that both service-defaults and
service-intentions (without destination wildcards) will ensure that the virtual
IP exists.
2023-05-31 15:40:06 -05:00
skpratt a065eef3ef
add FIPS to dataplane features (#17522) 2023-05-31 10:53:37 -05:00
Jared Kirschner b9c9d79778
Accept ap, datacenter, and namespace query params (#17525)
This commit only contains the OSS PR (datacenter query param support).
A separate enterprise PR adds support for ap and namespace query params.

Resources in Consul can exists within scopes such as datacenters, cluster
peers, admin partitions, and namespaces. You can refer to those resources from
interfaces such as the CLI, HTTP API, DNS, and configuration files.

Some scope levels have consistent naming: cluster peers are always referred to
as "peer".

Other scope levels use a short-hand in DNS lookups...
- "ns" for namespace
- "ap" for admin partition
- "dc" for datacenter

...But use long-hand in CLI commands:
- "namespace" for namespace
- "partition" for admin partition
- and "datacenter"

However, HTTP API query parameters do not follow a consistent pattern,
supporting short-hand for some scopes but long-hand for others:
- "ns" for namespace
- "partition" for admin partition
- and "dc" for datacenter.

This inconsistency is confusing, especially for users who have been exposed to
providing scope names through another interface such as CLI or DNS queries.

This commit improves UX by consistently supporting both short-hand and
long-hand forms of the namespace, partition, and datacenter scopes in HTTP API
query parameters.
2023-05-31 11:50:24 -04:00
skpratt fdda7adeaa
issue a warning if major FIPS assumptions are broken (#17524) 2023-05-31 09:01:44 -05:00
skpratt a46ac4be07
FIPS gossip changes (#17507)
* separate fips gossip

* clean up
2023-05-30 17:40:31 -05:00
skpratt e559c59eb6
Add version endpoint (#17506)
* add FIPS verison info

* separate out feature functionality from build identification

* split out ent test

* add version endpoint
2023-05-30 17:25:48 -05:00
Dhia Ayachi 04a0d0133a
fix isServer to exclude local address (#17519) 2023-05-30 15:31:07 -04:00
Eric Haberkorn d99312b86e
Add Upstream Service Targeting to Property Override Extension (#17517)
* add upstream service targeting to property override extension

* Also add baseline goldens for service specific property override extension.
* Refactor the extension framework to put more logic into the templates.

* fix up the golden tests
2023-05-30 14:53:42 -04:00
Nick Ethier 44f90132e0
hoststats: add package for collecting host statistics including cpu memory and disk usage (#17038) 2023-05-30 18:43:29 +00:00
Ashvitha 85cfec6b16
Add safety checks for the client telemetry gateway payload in case it's down (#17511) 2023-05-30 14:26:09 -04:00
Ronald 55e283dda9
[NET-3092] JWT Verify claims handling (#17452)
* [NET-3092] JWT Verify claims handling
2023-05-30 13:38:33 -04:00
Chris Thain 65b8ccdc1b
Enable Network filters for Wasm Envoy Extension (#17505) 2023-05-30 07:17:33 -07:00
Ashvitha 091925bcb7
HCP Telemetry Feature (#17460)
* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* [HCP Observability] Init OTELSink in Telemetry (#17162)

* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Initialize OTELSink with sync.Map for all the instrument stores.

* Added telemetry agent to client and init sink in deps

* Fixed client

* Initalize sink in deps

* init sink in telemetry library

* Init deps before telemetry

* Use concrete telemetry.OtelSink type

* add /v1/metrics

* Avoid returning err for telemetry init

* move sink init within the IsCloudEnabled()

* Use HCPSinkOpts in deps instead

* update golden test for configuration file

* Switch to using extra sinks in the telemetry library

* keep name MetricsConfig

* fix log in verifyCCMRegistration

* Set logger in context

* pass around MetricSink in deps

* Fix imports

* Rebased onto otel sink pr

* Fix URL in test

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* pass extraSinks as function param instead

* Add default interval as package export

* remove verifyCCM func

* Add clusterID

* Fix import and add t.Parallel() for missing tests

* Kick Vercel CI

* Remove scheme from endpoint path, and fix error logging

* return metrics.MetricSink for sink method

* Update SDK

* [HCP Observability] Metrics filtering and Labels in Go Metrics sink (#17184)

* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Initialize OTELSink with sync.Map for all the instrument stores.

* Added telemetry agent to client and init sink in deps

* Fixed client

* Initalize sink in deps

* init sink in telemetry library

* Init deps before telemetry

* Use concrete telemetry.OtelSink type

* add /v1/metrics

* Avoid returning err for telemetry init

* move sink init within the IsCloudEnabled()

* Use HCPSinkOpts in deps instead

* update golden test for configuration file

* Switch to using extra sinks in the telemetry library

* keep name MetricsConfig

* fix log in verifyCCMRegistration

* Set logger in context

* pass around MetricSink in deps

* Fix imports

* Rebased onto otel sink pr

* Fix URL in test

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* pass extraSinks as function param instead

* Add default interval as package export

* remove verifyCCM func

* Add clusterID

* Fix import and add t.Parallel() for missing tests

* Kick Vercel CI

* Remove scheme from endpoint path, and fix error logging

* return metrics.MetricSink for sink method

* Update SDK

* Added telemetry agent to client and init sink in deps

* Add node_id and __replica__ default labels

* add function for default labels and set x-hcp-resource-id

* Fix labels tests

* Commit suggestion for getDefaultLabels

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>

* Fixed server.id, and t.Parallel()

* Make defaultLabels a method on the TelemetryConfig object

* Rename FilterList to lowercase filterList

* Cleanup filter implemetation by combining regex into a single one, and making the type lowercase

* Fix append

* use regex directly for filters

* Fix x-resource-id test to use mocked value

* Fix log.Error formats

* Forgot the len(opts.Label) optimization)

* Use cfg.NodeID instead

---------

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>

* remove replic tag (#17484)

* [HCP Observability] Add custom metrics for OTEL sink, improve logging, upgrade modules and cleanup metrics client (#17455)

* Add custom metrics for Exporter and transform operations

* Improve deps logging

Run go mod tidy

* Upgrade SDK and OTEL

* Remove the partial success implemetation and check for HTTP status code in metrics client

* Add x-channel

* cleanup logs in deps.go based on PR feedback

* Change to debug log and lowercase

* address test operation feedback

* use GetHumanVersion on version

* Fix error wrapping

* Fix metric names

* [HCP Observability] Turn off retries for now until dynamically configurable (#17496)

* Remove retries for now until dynamic configuration is possible

* Clarify comment

* Update changelog

* improve changelog

---------

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>
2023-05-29 16:11:08 -04:00
Michael Zalimeni e1df0f28bd
Support `Listener` and `ClusterLoadAssignment` in `property-override` (#17497)
* Support Listener in Property Override

Add support for patching `Listener` resources via the builtin
`property-override` extension.

Refactor existing listener patch code in `BasicEnvoyExtender` to
simplify addition of resource support.

* Support ClusterLoadAssignment in Property Override

Add support for patching `ClusterLoadAssignment` resources via the
builtin `property-override` extension.
2023-05-29 09:42:35 -04:00
Michael Zalimeni 5a46a8c604
Add `builtin/property-override` Envoy Extension (#17487)
`property-override` is an extension that allows for arbitrarily
patching Envoy resources based on resource matching filters. Patch
operations resemble a subset of the JSON Patch spec with minor
differences to facilitate patching pre-defined (protobuf) schemas.

See Envoy Extension product documentation for more details.

Co-authored-by: Eric Haberkorn <eric.haberkorn@hashicorp.com>
Co-authored-by: Kyle Havlovitz <kyle@hashicorp.com>
2023-05-26 19:52:09 +00:00
Chris Thain 516eb4febc
Add `builtin/ext-authz` Envoy Extension (#17495) 2023-05-26 12:22:54 -07:00
Chris Thain 2740d12d44
ENT->OSS merge for Consolidate `ListEnvoyExtender` into `BasicEnvoyExtender` (#17491) 2023-05-26 11:10:31 -07:00
Lincoln Stoll 3605fde865
perf: Remove expensive reflection from raft/mesh hot path (#16552)
* perf: Remove expensive reflection from raft/mesh hot path

Replaces a reflection-based copy of a struct in the mesh topology with a
deep-copy generated implementation.

This is in the hot-path of raft FSM updates, and the reflection overhead was a
substantial part of mesh registration times (~90%). This could manifest as raft
thread saturation, and resulting instability.

Co-authored-by: Joel Brandhorst <joel.brandhorst@gmail.com>

* add changelog

---------

Co-authored-by: Joel Brandhorst <joel.brandhorst@gmail.com>
Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-05-26 11:42:05 -06:00
Eric Haberkorn 17a280d51b
This fixes an issue where TCP services that are exported cannot be configured to failover. (#17469)
This will likely happen frequently with sameness groups. Relaxing this
constraint is harmless for failover because xds/endpoints exludes cross
partition and peer endpoints.
2023-05-25 12:50:20 -04:00
Eric Haberkorn 1c80892717
fix tproxy sameness groups (#17468) 2023-05-25 12:18:55 -04:00
sarahalsmiller b147323fb0
xds: Remove APIGateway ToIngress function (#17453)
* xds generation for routes api gateway

* Update gateway.go

* move buildHttpRoute into xds package

* Update agent/consul/discoverychain/gateway.go

* remove unneeded function

* convert http route code to only run for http protocol to future proof code path

* Update agent/consul/discoverychain/gateway.go

Co-authored-by: Mike Morris <mikemorris@users.noreply.github.com>

* fix tests, clean up http check logic

* clean up todo

* Fix casing in docstring

* Fix import block, adjust docstrings

* Rename func

* Consolidate docstring onto single line

* Remove ToIngress() conversion for APIGW, which generates its own xDS now

* update name and comment

* use constant value

* use constant

* rename readyUpstreams to readyListeners to better communicate what that function is doing

---------

Co-authored-by: Mike Morris <mikemorris@users.noreply.github.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-05-25 15:16:37 +00:00
sarahalsmiller 6d35edc21c
xds: generate routes directly from API gateway snapshot (#17392)
* xds generation for routes api gateway

* Update gateway.go

* move buildHttpRoute into xds package

* Update agent/consul/discoverychain/gateway.go

* remove unneeded function

* convert http route code to only run for http protocol to future proof code path

* Update agent/consul/discoverychain/gateway.go

Co-authored-by: Mike Morris <mikemorris@users.noreply.github.com>

* fix tests, clean up http check logic

* clean up todo

* Fix casing in docstring

* Fix import block, adjust docstrings

* update name and comment

* use constant value

* use constant

---------

Co-authored-by: Mike Morris <mikemorris@users.noreply.github.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-05-25 09:54:55 -05:00
Derek Menteer a90c9ce2b0
Fix ACL check on health endpoint (#17424)
Fix ACL check on health endpoint

Prior to this change, the service health API would not explicitly return an
error whenever a token with invalid permissions was given, and it would instead
return empty results.  With this change, a "Permission denied" error is returned
whenever data is queried. This is done to better support the agent cache, which
performs a fetch backoff sleep whenever ACL errors are encountered.  Affected
endpoints are: `/v1/health/connect/` and `/v1/health/ingress/`.
2023-05-24 16:35:55 -05:00
Derek Menteer e2f15cfe56
Fix namespaced peer service updates / deletes. (#17456)
* Fix namespaced peer service updates / deletes.

This change fixes a function so that namespaced services are
correctly queried when handling updates / deletes. Prior to this
change, some peered services would not correctly be un-exported.

* Add changelog.
2023-05-24 16:32:45 -05:00
Paul Glass 07ff9d3d64
Use original_dst filter instead of use_original_dst field (#17433) 2023-05-24 12:01:17 -05:00
Ronald ddb25cec0e
[NET-3092] Improve jwt-provider tests (#17430)
* [NET-3092] more tests, prior to verify claims work
2023-05-24 10:30:48 -04:00
Dan Stough d935c7b466
[OSS] gRPC Blocking Queries (#17426)
* feat: initial grpc blocking queries

* changelog and docs update
2023-05-23 17:29:10 -04:00
Dhia Ayachi f526dfd0ac
add necessary plumbing to implement per server ip based rate limiting (#17436) 2023-05-23 15:37:01 -04:00
R.B. Boyer 304d641fb1
extract some config entry helpers into package (#17434) 2023-05-23 12:15:30 -05:00
Paul Glass 7f4fd2735a
Only synthesize anonymous token in primary DC (#17231)
* Only synthesize anonymous token in primary DC
* Add integration test for wan fed issue
2023-05-23 09:38:04 -05:00
Michael Zalimeni b8d2640429
Disable remote proxy patching except AWS Lambda (#17415)
To avoid unintended tampering with remote downstreams via service
config, refactor BasicEnvoyExtender and RuntimeConfig to disallow
typical Envoy extensions from being applied to non-local proxies.

Continue to allow this behavior for AWS Lambda and the read-only
Validate builtin extensions.

Addresses CVE-2023-2816.
2023-05-23 11:55:06 +00:00
sarahalsmiller e2a81aa8bd
xds: generate listeners directly from API gateway snapshot (#17398)
* API Gateway XDS Primitives, endpoints and clusters (#17002)

* XDS primitive generation for endpoints and clusters

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* server_test

* deleted extra file

* add missing parents to test

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Routes for API Gateway (#17158)

* XDS primitive generation for endpoints and clusters

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* server_test

* deleted extra file

* add missing parents to test

* checkpoint

* delete extra file

* httproute flattening code

* linting issue

* so close on this, calling for tonight

* unit test passing

* add in header manip to virtual host

* upstream rebuild commented out

* Use consistent upstream name whether or not we're rebuilding

* Start working through route naming logic

* Fix typos in test descriptions

* Simplify route naming logic

* Simplify RebuildHTTPRouteUpstream

* Merge additional compiled discovery chains instead of overwriting

* Use correct chain for flattened route, clean up + add TODOs

* Remove empty conditional branch

* Restore previous variable declaration

Limit the scope of this PR

* Clean up, improve TODO

* add logging, clean up todos

* clean up function

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* checkpoint, skeleton, tests not passing

* checkpoint

* endpoints xds cluster configuration

* resources test fix

* fix reversion in resources_test

* checkpoint

* Update agent/proxycfg/api_gateway.go

Co-authored-by: John Maguire <john.maguire@hashicorp.com>

* unit tests passing

* gofmt

* add deterministic sorting to appease the unit test gods

* remove panic

* Find ready upstream matching listener instead of first in list

* Clean up, improve TODO

* Modify getReadyUpstreams to filter upstreams by listener (#17410)

Each listener would previously have all upstreams from any route that bound to the listener. This is problematic when a route bound to one listener also binds to other listeners and so includes upstreams for multiple listeners. The list for a given listener would then wind up including upstreams for other listeners.

* clean up todos, references to api gateway in listeners_ingress

* merge in Nathan's fix

* Update agent/consul/discoverychain/gateway.go

* cleanup current todos, remove snapshot manipulation from generation code

* Update agent/structs/config_entry_gateways.go

Co-authored-by: Thomas Eckert <teckert@hashicorp.com>

* Update agent/consul/discoverychain/gateway.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Update agent/consul/discoverychain/gateway.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Update agent/proxycfg/snapshot.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* clarified header comment for FlattenHTTPRoute, changed RebuildHTTPRouteUpstream to BuildHTTPRouteUpstream

* simplify cert logic

* Delete scratch

* revert route related changes in listener PR

* Update agent/consul/discoverychain/gateway.go

* Update agent/proxycfg/snapshot.go

* clean up uneeded extra lines in endpoints

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Thomas Eckert <teckert@hashicorp.com>
2023-05-22 17:36:29 -04:00
R.B. Boyer e00280e7df
prototest: fix early return condition in AssertElementsMatch (#17416) 2023-05-22 13:49:50 -05:00
sarahalsmiller d34bde0e4e
xds: generate clusters directly from API gateway snapshot (#17391)
* endpoints xds cluster configuration

* clusters xds native generation

* resources test fix

* fix reversion in resources_test

* Update agent/proxycfg/api_gateway.go

Co-authored-by: John Maguire <john.maguire@hashicorp.com>

* gofmt

* Modify getReadyUpstreams to filter upstreams by listener (#17410)

Each listener would previously have all upstreams from any route that bound to the listener. This is problematic when a route bound to one listener also binds to other listeners and so includes upstreams for multiple listeners. The list for a given listener would then wind up including upstreams for other listeners.

* Update agent/proxycfg/api_gateway.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Restore import blocking

* Undo removal of unrelated code

---------

Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-05-22 12:00:13 -04:00
Matt Keeler 93bad3ea1b
Allow resource updates to omit an owner refs UID (#17423)
This change enables workflows where you are reapplying a resource that should have an owner ref to publish modifications to the resources data without performing a read to figure out the current owner resource incarnations UID.

Basically we want workflows similar to `kubectl apply` or `consul config write` to be able to work seamlessly even for owned resources.

In these cases the users intention is to have the resource owned by the “current” incarnation of the owner resource.
2023-05-22 10:44:49 -04:00
Ronald 113202d541
JWT Authentication with service intentions: xds package update (#17414)
* JWT Authentication with service intentions: update xds package to translate config to envoy
2023-05-19 18:14:16 -04:00
sarahalsmiller 134aac7c26
xds: generate endpoints directly from API gateway snapshot (#17390)
* endpoints xds cluster configuration

* resources test fix

* fix reversion in resources_test

* Update agent/proxycfg/api_gateway.go

Co-authored-by: John Maguire <john.maguire@hashicorp.com>

* gofmt

* Modify getReadyUpstreams to filter upstreams by listener (#17410)

Each listener would previously have all upstreams from any route that bound to the listener. This is problematic when a route bound to one listener also binds to other listeners and so includes upstreams for multiple listeners. The list for a given listener would then wind up including upstreams for other listeners.

* Update agent/proxycfg/api_gateway.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Restore import blocking

* Skip to next route if route has no upstreams

* cleanup

* change set from bool to empty struct

---------

Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-05-19 18:50:59 +00:00
Matt Keeler 1d6a0c8f21
Add the workload health controller (#17215) 2023-05-19 13:53:29 -04:00
Kyle Havlovitz 2904d0a431
Pull virtual IPs for filter chains from discovery chains (#17375) 2023-05-17 11:18:39 -07:00