Commit Graph

173 Commits

Author SHA1 Message Date
Matt Keeler a7774a9538
Introduce randomized timings and reproducible randomization into controller integration tests. (#19393)
As the V2 architecture hinges on eventual consistency and controllers reconciling the existing state in response to writes, there are potential issues we could run into regarding ordering and timing of operations. We want to be able to guarantee that given a set of resources the system will always eventually get to the desired correct state. The order of resource writes and delays in performing those writes should not alter the final outcome of reaching the desired state.

To that end, this commit introduces arbitrary randomized delays before performing resources writes into the `resourcetest.Client`. Its `PublishResources` method was already randomizing the order of resource writes. By default, no delay is added to normal writes and deletes but tests can opt-in via either passing hard coded options when creating the `resourcetest.Client` or using the `resourcetest.ConfigureTestCLIFlags` function to allow processing of CLI parameters.

In addition to allowing configurability of the request delay min and max, the client also has a configurable random number generator seed. When Using the CLI parameter helpers, a test log will be written noting the currently used settings. If the test fails then you can reproduce the same delays and order randomizations by providing the seed during the previous test failure.
2023-11-08 10:45:25 -05:00
cskh 38d94282ca
Ci upgrade test 1 17 (#19536)
CI: upgrade test from 1.17
2023-11-06 21:35:59 +00:00
R.B. Boyer 65592d91a8
chore: apply enterprise changes that were missed to some testing files (#19504)
This should align between CE ef35525 and ENT 7f95226dbe40151c8f17dd4464784b60cf358dc1 in:

- testing/integration/consul-container
- test-integ
- testing/deployer
2023-11-03 11:43:43 -05:00
Dan Stough fd128f4947
build: dependency updates for 1.17.0 (#19453) 2023-11-03 12:29:59 -04:00
Semir Patel ef35525cf1
resource: finalizer aware delete endpoint (2 of 5) (#19493)
resource: make delete endpoint finalizer aware
2023-11-03 10:10:58 -04:00
R.B. Boyer 4b85aa5a97
testing/deployer: support tproxy in v2 for dataplane (#19094)
This updates the testing/deployer (aka "topology test") framework to allow for a 
v2-oriented topology to opt services into enabling TransparentProxy. The restrictions 
are similar to that of #19046

The multiport Ports map that was added in #19046 was changed to allow for the 
protocol to be specified at this time, but for now the only supported protocol is TCP 
as only L4 functions currently on main.

As part of making transparent proxy work, the DNS server needed a new zonefile 
for responding to virtual.consul requests, since there is no Kubernetes DNS and 
the Consul DNS work for v2 has not happened yet. Once Consul DNS supports v2 we should switch over. For now the format of queries is:

<service>--<namespace>--<partition>.virtual.consul

Additionally:

- All transparent proxy enabled services are assigned a virtual ip in the 10.244.0/24
  range. This is something Consul will do in v2 at a later date, likely during 1.18.
- All services with exposed ports (non-mesh) are assigned a virtual port number for use
  with tproxy
- The consul-dataplane image has been made un-distroless, and gotten the necessary
  tools to execute consul connect redirect-traffic before running dataplane, thus simulating
  a kubernetes init container in plain docker.
2023-11-02 16:13:16 -05:00
Semir Patel aaac20f4a8
resource: misc finalizer apis (#19474) 2023-11-02 15:56:02 -05:00
R.B. Boyer a72f868218
testing/deployer: update deployer to use v2 catalog constructs when requested (#19046)
This updates the testing/deployer (aka "topology test") framework to conditionally 
configure and launch catalog constructs using v2 resources. This is controlled via a 
Version field on the Node construct in a topology.Config. This only functions for a 
dataplane type and has other restrictions that match the rest of v2 (no peering, no 
wanfed, no mesh gateways).

Like config entries, you can statically provide a set of initial resources to be synced 
when bringing up the cluster (beyond those that are generated for you such as 
workloads, services, etc).

If you want to author a test that can be freely converted between v1 and v2 then that 
is possible. If you switch to the multi-port definition on a topology.Service (aka 
"workload/instance") then that makes v1 ineligible.

This also adds a starter set of "on every PR" integration tests for single and multiport 
under test-integ/catalogv2
2023-11-02 14:25:48 -05:00
Derek Menteer 8f4c43727d
[NET-5916] Fix locality-aware routing config and tests (CE) (#19483)
Fix locality-aware routing config and tests
2023-11-02 14:05:06 -05:00
Michael Zalimeni 42647de35d
[NET-6138] security: Bump `google.golang.org/grpc` to 1.56.3 (CVE-2023-44487) (#19414)
Bump google.golang.org/grpc to 1.56.3

This resolves [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487).

Co-authored-by: Chris Thain <chris.m.thain@gmail.com>
2023-10-30 08:44:22 -04:00
John Murret f0cf8f2f40
NET-6294 - v1 Agentless proxycfg datasource errors after v2 changes (#19365) 2023-10-27 14:06:38 -06:00
Dhia Ayachi 12ef115b61
bump raft-wal version to 0.4.1 (#19314)
* bump raft-wal version to 0.4.1

* changelog

* go mod tidy integration tests

* go mod tidy test-integ
2023-10-24 10:47:46 -04:00
Michael Zalimeni 8eb074e7c1
[NET-5944] security: Update Go version to 1.20.10 and `x/net` to 0.17.0 (#19225)
* Bump golang.org/x/net to 0.17.0

This resolves [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487).

* Update Go version to 1.20.10

This resolves [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487)
(`net/http`).
2023-10-16 17:49:04 -04:00
Semir Patel 4c5a46e5e1
v2tenancy: rename v1alpha1 -> v2beta1 (#19227) 2023-10-16 21:43:47 +00:00
R.B. Boyer c26d5cf62c
test: fix container test enterprise drift (#19101) 2023-10-11 15:39:09 -05:00
Dhia Ayachi ab1e08f1a4
fix flaking container tests (#19134) 2023-10-11 11:26:07 -04:00
Chris S. Kim 92ce814693
Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
Eric Haberkorn ad3aab1ef7
Add traffic permissions integration tests. (#19008)
Add traffic permissions integration tests.
2023-10-06 12:06:12 -04:00
Eric Haberkorn 9656fd157f
Fix Explicit Destination Integration Test (#19060)
fix explicit destination integration test
2023-10-04 12:52:12 -04:00
Eric Haberkorn f2b7b4591a
Fix Traffic Permissions Default Deny (#19028)
Whenver a traffic permission exists for a given workload identity, turn on default deny.

Previously, this was only working at the port level.
2023-10-04 09:58:28 -04:00
Poonam Jadhav 6c92dd1359
NET-5600/container-test-acl-enabled (#18887)
* feat: add container tests for resource http api with acl enabled

* refactor: clean up
2023-10-03 10:55:31 -04:00
Matt Keeler 2240e746dd
Skip the catalog v2 upgrade test (#19005)
We intentionally broke api compatibility here as we are not yet maintaining backwards compat for the v2 apis
2023-09-26 17:05:19 -04:00
John Murret 6f0df20d10
NET-5657 - consul-containers test for explicit upstreams (#18952)
* Explicit container test

* remove static resources

* fix passing serviceBindPorts

* WIP

* fix explicit upstream test

* use my image in CI until dataplane is fixed.

* gofmt

* fixing reference to v2beta1 in test-containers

* WIP

* remove bad references

* add missing license headers

* allow access internal/resource/resourcetest

* fix check-allowed-imports to append array items

* use preview image for dataplane

* revert some inadverntent comment updates in peering_topology

* add building local consul-dataplane image to compatibility-tests CI

* fix substitution in CI

* change upstreams to destinations based on incoming change

* fixing use of upstreams in resource update

* remove commented out lines and enable envoy concurrency on dataplane.

* changes to addess PR feedback

* small fixes

---------

Co-authored-by: Eric <eric@haberkorn.co>
2023-09-26 16:21:47 -04:00
Iryna Shustava d88888ee8b
catalog,mesh,auth: Bump versions to v2beta1 (#18930) 2023-09-22 10:51:15 -06:00
Curt Bushko bc142cd152
NET-4884 - Terminating gateway tests for namespaces & partitions (#18820)
* Add gateway test to CE
2023-09-21 10:25:27 -04:00
John Landa 9eaa8eb026
dns token (#17936)
* dns token

fix whitespace for docs and comments

fix test cases

fix test cases

remove tabs in help text

Add changelog

Peering dns test

Peering dns test

Partial implementation of Peered DNS test

Swap to new topology lib

expose dns port for integration tests on client

remove partial test implementation

remove extra port exposure

remove changelog from the ent pr

Add dns token to set-agent-token switch

Add enterprise golden file

Use builtin/dns template in tests

Update ent dns policy

Update ent dns template test

remove local gen certs

fix templated policy specs

* add changelog

* go mod tidy
2023-09-20 15:50:06 -06:00
Chris Thain a071899684
Add otel-access-logging Envoy extension integration test (#18898) 2023-09-19 19:04:47 +00:00
Ashesh Vidyut 6fd33ba30d
NET-4519 Collecting journald logs in "consul debug" bundle (#18797)
* debug since

* fix docs

* chagelog added

* fix go mod

* debug test fix

* fix test

* tabs test fix

* Update .changelog/18797.txt

Co-authored-by: Ganesh S <ganesh.seetharaman@hashicorp.com>

---------

Co-authored-by: Ganesh S <ganesh.seetharaman@hashicorp.com>
2023-09-19 08:46:50 +05:30
sarahalsmiller 753c8f1774
Retry and timeout test acceptance test (#18791)
* retry and timeout test

* add docker mirrior

* checkpoint

* add in error

* add in delay

* up error rate

* fix status code
2023-09-15 11:54:51 -05:00
Nathan Coleman e5d26a13cd
NET-5530 Support response header modifiers on http-route config entry (#18646)
* Add response header filters to http-route config entry definitions

* Map response header filters from config entry when constructing route destination

* Support response header modifiers at the service level as well

* Update protobuf definitions

* Update existing unit tests

* Add response filters to route consolidation logic

* Make existing unit tests more robust

* Add missing docstring

* Add changelog entry

* Add response filter modifiers to existing integration test

* Add more robust testing for response header modifiers in the discovery chain

* Add more robust testing for request header modifiers in the discovery chain

* Modify test to verify that service filter modifiers take precedence over rule filter modifiers
2023-09-08 14:04:56 -04:00
Ronald bbef879f85
[NET-5325] ACL templated policies support in tokens and roles (#18708)
* [NET-5325] ACL templated policies support in tokens and roles
- Add API support for creating tokens/roles with templated-policies
- Add CLI support for creating tokens/roles with templated-policies

* adding changelog
2023-09-08 12:45:24 +00:00
Iryna Shustava 3c70e14713
sidecar-proxy controller: L4 controller with explicit upstreams (NET-3988) (#18352)
* This controller generates and saves ProxyStateTemplate for sidecar proxies.
* It currently supports single-port L4 ports only.
* It keeps a cache of all destinations to make it easier to compute and retrieve destinations.
* It will update the status of the pbmesh.Upstreams resource if anything is invalid.
* This commit also changes service endpoints to include workload identity. This made the implementation a bit easier as we don't need to look up as many workloads and instead rely on endpoints data.
2023-09-07 09:37:15 -06:00
Ashvitha 0f48b7af5e
[HCP Telemetry] Move first TelemetryConfig Fetch into the TelemetryConfigProvider (#18318)
* Add Enabler interface to turn sink on/off

* Use h for hcpProviderImpl vars, fix PR feeback and fix errors

* Keep nil check in exporter and fix tests

* Clarify comment and fix function name

* Use disable instead of enable

* Fix errors nit in otlp_transform

* Add test for refreshInterval of updateConfig

* Add disabled field in MetricsConfig struct

* Fix PR feedback: improve comment and remove double colons

* Fix deps test which requires a maybe

* Update hcp-sdk-go to v0.61.0

* use disabled flag in telemetry_config.go

* Handle 4XX errors in telemetry_provider

* Fix deps test

* Check 4XX instead

* Run make go-mod-tidy
2023-08-30 13:25:26 -04:00
Semir Patel 53e28a4963
OSS -> CE (community edition) changes (#18517) 2023-08-22 09:46:03 -05:00
Michael Zalimeni 61b7c0d76f
[NET-5163] Support locality testing in consul-container (#18484)
* Support locality testing in consul-container

Support including locality in client sidecar config.

Also align test config structs with Ent to avoid future conflicts.

* Refactor consul-container fortio helpers

Refactor fortio test helpers to separate HTTP retries from waiting on
fortio result changes due to e.g. service startup and failovers.
2023-08-17 09:41:29 -04:00
Anita Akaeze 0e94f48ce0
NET-5187: Upgrade test timeout due to log producer errors (#18461)
Stop log producer before restarting container
2023-08-15 12:34:02 -04:00
hashicorp-copywrite[bot] 5fb9df1640
[COMPLIANCE] License changes (#18443)
* Adding explicit MPL license for sub-package

This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.

* Adding explicit MPL license for sub-package

This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.

* Updating the license from MPL to Business Source License

Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at <Blog URL>, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.

* add missing license headers

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

* Update copyright file headers to BUSL-1.1

---------

Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
2023-08-11 09:12:13 -04:00
Dan Stough 948ce8bc23
build: updates for 1.16.1 release (#18415)
* build: updates for 1.16.1 release

* build: fix missing replace directive for envoyextensions

* build: go mod tidy testing/deployer
2023-08-09 17:12:34 -04:00
cskh 43d8898e08
bump testcontainers-go from 0.22.0 and remove pinned go version in in… (#18395)
* bump testcontainers-go from 0.22.0 and remove pinned go version in integ test

* go mod tidy

* Replace deprecated target.Authority with target.URL.Host
2023-08-08 18:08:14 +00:00
cskh 7902ae20a1
Upgrade test: remove outdated test and disable log due to verbosity (#18403)
* remove outdated test

* disable log since we have too many parallel tests
2023-08-08 13:16:32 -04:00
R.B. Boyer 48effe5f8a
chore: make go-mod-tidy (#18388) 2023-08-07 14:02:34 -05:00
Michael Zalimeni 905e371607
[NET-5146] security: Update Go version to 1.20.7 and `x/net` to 0.13.0 (#18358)
* Update Go version to 1.20.7

This resolves [CVE-2023-29409]
(https://nvd.nist.gov/vuln/detail/CVE-2023-29409)(`crypto/tls`).

* Bump golang.org/x/net to 0.13.0

Addresses [CVE-2023-3978](https://nvd.nist.gov/vuln/detail/CVE-2023-3978)
for security scans (non-impacting).
2023-08-02 13:10:29 -04:00
Ashvitha 828567c62e
[HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration (#18168)
* OTElExporter now uses an EndpointProvider to discover the endpoint

* OTELSink uses a ConfigProvider to obtain filters and labels configuration

* improve tests for otel_sink

* Regex logic is moved into client for a method on the TelemetryConfig object

* Create a telemetry_config_provider and update deps to use it

* Fix conversion

* fix import newline

* Add logger to hcp client and move telemetry_config out of the client.go file

* Add a telemetry_config.go to refactor client.go

* Update deps

* update hcp deps test

* Modify telemetry_config_providers

* Check for nil filters

* PR review updates

* Fix comments and move around pieces

* Fix comments

* Remove context from client struct

* Moved ctx out of sink struct and fixed filters, added a test

* Remove named imports, use errors.New if not fformatting

* Remove HCP dependencies in telemetry package

* Add success metric and move lock only to grab the t.cfgHahs

* Update hash

* fix nits

* Create an equals method and add tests

* Improve telemetry_config_provider.go tests

* Add race test

* Add missing godoc

* Remove mock for MetricsClient

* Avoid goroutine test panics

* trying to kick CI lint issues by upgrading mod

* imprve test code and add hasher for testing

* Use structure logging for filters, fix error constants, and default to allow all regex

* removed hashin and modify logic to simplify

* Improve race test and fix PR feedback by removing hash equals and avoid testing the timer.Ticker logic, and instead unit test

* Ran make go-mod-tidy

* Use errtypes in the test

* Add changelog

* add safety check for exporter endpoint

* remove require.Contains by using error types, fix structure logging, and fix success metric typo in exporter

* Fixed race test to have changing config values

* Send success metric before modifying config

* Avoid the defer and move the success metric under
2023-08-01 17:20:18 -04:00
John Landa 2a8bf5df61
Wasm integration tests for local and remote wasm files (#17756)
* wasm integration tests for local and remote wasm files

refactoring and cleanup for wasm testing

remove wasm debug logging

PR feedback, wasm build lock

correct path pattern for wasm build files

Add new helper function to minimize changes to existing test code

Remove extra param

mod tidy

add custom service setup to test lib

add wait until static server sidecar can reach nginx sidecar

Doc comments

PR feedback

Update workflows to compile wasm for integration tests

Fix docker build path

Fix package name for linter

Update makefile, fix redeclared function

Update expected wasm filename

Debug test ls in workflow

remove pwd in favor of relative path

more debugging

Build wasm in compatability tests as well

Build wasm directly in ci rather than in container

Debug tinygo and llvm version

Change wasm file extension

Remove tinygo debugging

Remove extra comments

* Add compiled wasm and build instructions
2023-08-01 15:49:39 -05:00
cskh c2066b9b22
NET-4222 take config file consul container (#18218)
Net 4222 take config file consul container
2023-07-20 21:30:22 -04:00
Michael Zalimeni 72999bb735
Update submodules to latest following 1.16.0 (#18197)
Align all our internal use of submodules on the latest versions.
2023-07-19 18:57:34 -04:00
Michael Zalimeni 29cdb75d05
[NET-4865] Bump golang.org/x/net to 0.12.0 (#18186)
Bump golang.org/x/net to 0.12.0

While not necessary to directly address CVE-2023-29406 (which should be
handled by using a patched version of Go when building), an
accompanying change to HTTP/2 error handling does impact agent code.

See https://go-review.googlesource.com/c/net/+/506995 for the HTTP/2
change.

Bump this dependency across our submodules as well for the sake of
potential indirect consumers of `x/net/http`.
2023-07-19 11:37:58 -04:00
Nick Irvine cd3fc9e1d0
add peering_commontopo tests [NET-3700] (#17951)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: NiniOak <anita.akaeze@hashicorp.com>
2023-07-18 16:41:30 -07:00
Ronald 921445712e
[NET-4792] Add integrations tests for jwt-auth (#18169) 2023-07-18 14:59:01 -04:00
Ronald 03cf37e7b8
Re-order expected/actual for assertContainerState in consul container tests (#18157)
Re-order expected/actual, consul container tests
2023-07-17 18:11:19 +00:00