consul/agent
John Murret e027c94b52
adding config for request_limits (#15531)
* server: add placeholder glue for rate limit handler

This commit adds a no-op implementation of the rate-limit handler and
adds it to the `consul.Server` struct and setup code.

This allows us to start working on the net/rpc and gRPC interceptors and
config logic.

* Add handler errors

* Set the global read and write limits

* fixing multilimiter moving packages

* Fix typo

* Simplify globalLimit usage

* add multilimiter and tests

* exporting LimitedEntity

* Apply suggestions from code review

Co-authored-by: John Murret <john.murret@hashicorp.com>

* add config update and rename config params

* add doc string and split config

* Apply suggestions from code review

Co-authored-by: Dan Upton <daniel@floppy.co>

* use timer to avoid go routine leak and change the interface

* add comments to tests

* fix failing test

* add prefix with config edge, refactor tests

* Apply suggestions from code review

Co-authored-by: Dan Upton <daniel@floppy.co>

* refactor to apply configs for limiters under a prefix

* add fuzz tests and fix bugs found. Refactor reconcile loop to have a simpler logic

* make KeyType an exported type

* split the config and limiter trees to fix race conditions in config update

* rename variables

* fix race in test and remove dead code

* fix reconcile loop to not create a timer on each loop

* add extra benchmark tests and fix tests

* fix benchmark test to pass value to func

* server: add placeholder glue for rate limit handler

This commit adds a no-op implementation of the rate-limit handler and
adds it to the `consul.Server` struct and setup code.

This allows us to start working on the net/rpc and gRPC interceptors and
config logic.

* Set the global read and write limits

* fixing multilimiter moving packages

* add server configuration for global rate limiting.

* remove agent test

* remove added stuff from handler

* remove added stuff from multilimiter

* removing unnecessary TODOs

* Removing TODO comment from handler

* adding in defaulting to infinite

* add disabled status in there

* adding in documentation for disabled mode.

* make disabled the default.

* Add mock and agent test

* addig documentation and missing mock file.

* Fixing test TestLoad_IntegrationWithFlags

* updating docs based on PR feedback.

* Updating Request Limits mode to use int based on PR feedback.

* Adding RequestLimits struct so we have a nested struct in ReloadableConfig.

* fixing linting references

* Update agent/consul/rate/handler.go

Co-authored-by: Dan Upton <daniel@floppy.co>

* Update agent/consul/config.go

Co-authored-by: Dan Upton <daniel@floppy.co>

* removing the ignore of the request limits in JSON.  addingbuilder logic to convert any read rate or write rate less than 0 to rate.Inf

* added conversion function to convert request limits object to handler config.

* Updating docs to reflect gRPC and RPC are rate limit and as a result, HTTP requests are as well.

* Updating values for TestLoad_FullConfig() so that they were different and discernable.

* Updating TestRuntimeConfig_Sanitize

* Fixing TestLoad_IntegrationWithFlags test

* putting nil check in place

* fixing rebase

* removing change for missing error checks.  will put in another PR

* Rebasing after default multilimiter config change

* resolving rebase issues

* updating reference for incomingRPCLimiter to use interface

* updating interface

* Updating interfaces

* Fixing mock reference

Co-authored-by: Daniel Upton <daniel@floppy.co>
Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
2022-12-13 13:09:55 -07:00
..
ae sdk: add TestLogLevel for setting log level in tests 2022-02-03 13:42:28 -05:00
auto-config removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
cache cache: refactor agent cache fetching to prevent unnecessary fetches on error (#14956) 2022-10-25 10:27:26 -05:00
cache-types Regenerate files according to 1.19.2 formatter 2022-10-24 16:12:08 -04:00
checks removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
config adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
configentry Fix mesh gateway configuration with proxy-defaults (#15186) 2022-11-09 10:14:29 -06:00
connect connect: use -dev-no-store-token for test vaults to reduce source of flakes (#15691) 2022-12-06 13:15:13 -06:00
consul adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
debug bulk rewrite using this script 2022-01-20 10:46:23 -06:00
dns test: fix incorrect use of t instead of r in retry test (#13146) 2022-05-19 14:00:07 -05:00
exec re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
grpc-external adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
grpc-internal adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
grpc-middleware adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
hcp removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
local Merge pull request #13499 from maxb/delete-unused-metric 2022-08-01 12:31:05 -06:00
metadata Change serf-tag references to field references. 2022-08-31 16:38:42 -05:00
mock
pool Use rpcHoldTimeout to calculate blocking timeout (#15541) 2022-11-24 10:13:02 -05:00
proxycfg feat(ingress-gateway): support outlier detection of upstream service for ingress gateway (#15614) 2022-12-13 11:51:37 -05:00
proxycfg-glue Fixup authz for data imported from peers (#15347) 2022-11-14 11:36:27 -07:00
proxycfg-sources chore: remove unused argument from MergeNodeServiceWithCentralConfig (#15024) 2022-11-09 14:54:57 +00:00
router Regenerate files according to 1.19.2 formatter 2022-10-24 16:12:08 -04:00
routine-leak-checker removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
rpc adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
rpcclient/health test: fix flaky TestHealthServiceNodes_NodeMetaFilter by waiting until the streaming subsystem has a valid grpc connection (#15019) 2022-10-24 13:09:53 -05:00
structs feat: add access logging API to proxy defaults (#15780) 2022-12-13 14:52:18 -05:00
submatview Update hcp-scada-provider to fix diamond dependency problem with go-msgpack (#15185) 2022-11-07 11:34:30 -05:00
systemd
token removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
uiserver removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
xds feat(ingress-gateway): support outlier detection of upstream service for ingress gateway (#15614) 2022-12-13 11:51:37 -05:00
acl.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
acl_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
acl_endpoint_legacy.go
acl_endpoint_legacy_test.go
acl_endpoint_test.go [OSS] Add new peering ACL rule (#13848) 2022-07-22 14:42:23 -06:00
acl_oss.go agent: support `X-Consul-Results-Filtered-By-ACLs` header in agent-local endpoints (#11610) 2021-12-03 20:36:28 +00:00
acl_test.go test: ensure that all dependencies in a test agent use the test logger (#14996) 2022-10-24 17:02:38 -05:00
agent.go adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
agent_endpoint.go Merge branch 'main' of github.com:hashicorp/consul into derekm/split-grpc-ports 2022-09-08 14:53:08 -05:00
agent_endpoint_oss.go Fixup acl.EnterpriseMeta 2022-04-05 15:11:49 -07:00
agent_endpoint_oss_test.go Add oss test 2022-05-09 10:07:19 -07:00
agent_endpoint_test.go connect: ensure all vault connect CA tests use limited privilege tokens (#15669) 2022-12-06 10:06:36 -06:00
agent_oss.go proxycfg: replace direct agent cache usage with interfaces (#13320) 2022-06-01 16:18:06 +01:00
agent_test.go adding config for request_limits (#15531) 2022-12-13 13:09:55 -07:00
apiserver.go
apiserver_test.go
catalog_endpoint.go [OSS] Support merge-central-config option in node services list API (#13450) 2022-06-15 08:30:31 -07:00
catalog_endpoint_oss.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
catalog_endpoint_test.go block PeerName register requests (#13887) 2022-07-29 14:36:22 -07:00
check.go Fixup acl.EnterpriseMeta 2022-04-05 15:11:49 -07:00
config_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
config_endpoint_test.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
connect_auth.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
connect_ca_endpoint.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
connect_ca_endpoint_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
coordinate_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
coordinate_endpoint_test.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
delegate_mock_test.go Move ACLResolveResult into acl/resolver package (#13467) 2022-06-17 10:24:43 +01:00
denylist.go
denylist_test.go
discovery_chain_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
discovery_chain_endpoint_test.go Cluster peering failover disco chain changes (#14296) 2022-08-23 09:13:43 -04:00
dns.go Add peering `.service` and `.node` DNS lookups. (#15596) 2022-11-29 12:23:18 -06:00
dns_oss.go Add peering `.service` and `.node` DNS lookups. (#15596) 2022-11-29 12:23:18 -06:00
dns_oss_test.go Add peering `.service` and `.node` DNS lookups. (#15596) 2022-11-29 12:23:18 -06:00
dns_test.go test: remove variable shadowing in TestDNS_ServiceLookup_ARecordLimits (#15740) 2022-12-09 10:19:02 -06:00
enterprise_delegate_oss.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
event_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
event_endpoint_test.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
federation_state_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
health_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
health_endpoint_test.go test: fix flaky TestHealthServiceNodes_NodeMetaFilter by waiting until the streaming subsystem has a valid grpc connection (#15019) 2022-10-24 13:09:53 -05:00
http.go Allow consul debug on non-ACL consul servers (#15155) 2022-10-27 09:25:18 -04:00
http_decode_test.go Regenerate files according to 1.19.2 formatter 2022-10-24 16:12:08 -04:00
http_oss.go Unify various status errors into one HTTP error type. (#12594) 2022-04-29 13:42:49 -04:00
http_oss_test.go Remove references to "master" ACL tokens in tests (#11751) 2021-12-07 12:48:50 +00:00
http_register.go Leadership transfer cmd (#14132) 2022-11-14 15:35:12 -05:00
http_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
intentions_endpoint.go oss: peering, http: get peer service intentions (#2098) 2022-06-22 16:25:09 -07:00
intentions_endpoint_oss_test.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
intentions_endpoint_test.go test: possibly fix flake in TestIntentionGetExact (#15021) 2022-10-18 10:51:20 -05:00
keyring.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
keyring_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
kvs_endpoint.go Fix KVSGet method to handle QueryOptions properly (#13344) 2022-06-02 12:26:18 -04:00
kvs_endpoint_test.go
metrics.go
metrics_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
nodeid.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
nodeid_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
notify.go
notify_test.go
operator_endpoint.go Leadership transfer cmd (#14132) 2022-11-14 15:35:12 -05:00
operator_endpoint_oss.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
operator_endpoint_test.go
peering_endpoint.go Support Stale Queries for Trust Bundle Lookups (#14724) 2022-09-28 09:56:59 -07:00
peering_endpoint_oss_test.go [sync oss] api: add peering api module (#12911) 2022-05-02 11:49:05 -07:00
peering_endpoint_test.go Bring back parameter ServerExternalAddresses in GenerateToken endpoint (#15267) 2022-11-08 14:55:18 -06:00
prepared_query_endpoint.go Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
prepared_query_endpoint_test.go Add Cluster Peering Failover Support to Prepared Queries (#13835) 2022-07-22 09:14:43 -04:00
reload.go
remote_exec.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
remote_exec_test.go Remove references to "master" ACL tokens in tests (#11751) 2021-12-07 12:48:50 +00:00
retry_join.go add HCP integration component (#14723) 2022-09-26 14:58:15 -04:00
retry_join_test.go add HCP integration component (#14723) 2022-09-26 14:58:15 -04:00
service_checks_test.go Service http checks data source for agentless proxies (#14924) 2022-10-12 07:49:56 -07:00
service_manager.go Merge central config for GetEnvoyBootstrapParams (#14869) 2022-10-10 12:40:27 -05:00
service_manager_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
session_endpoint.go fix: clarifying error message when acquiring a lock in remote dc (#15394) 2022-11-16 15:27:37 -05:00
session_endpoint_test.go
setup.go Use rpcHoldTimeout to calculate blocking timeout (#15541) 2022-11-24 10:13:02 -05:00
setup_oss.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
sidecar_service.go PR #14057 follow up fix: service id parsing from sidecar id (#14541) 2022-09-09 09:47:10 -05:00
sidecar_service_test.go Backport test from ENT: "Fix missing test fields" (#15258) 2022-11-04 09:29:16 -05:00
signal_unix.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
signal_windows.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
snapshot_endpoint.go
snapshot_endpoint_test.go
status_endpoint.go
status_endpoint_test.go
streaming_test.go
testagent.go cache: refactor agent cache fetching to prevent unnecessary fetches on error (#14956) 2022-10-25 10:27:26 -05:00
testagent_test.go
translate_addr.go
txn_endpoint.go Merge pull request #13388 from deblasis/feature/health-checks_windows_service 2022-10-17 09:26:19 -04:00
txn_endpoint_test.go increase the size of txn to support vault (#14599) 2022-09-19 09:07:19 -07:00
ui_endpoint.go Add new internal endpoint to list exported services to a peer 2022-09-23 09:43:56 -04:00
ui_endpoint_oss_test.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
ui_endpoint_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00
user_event.go Vendor in rpc mono repo for net/rpc fork, go-msgpack, msgpackrpc. (#12311) 2022-02-14 09:45:45 -08:00
user_event_test.go chore: removed unused method AddService (#13905) 2022-07-26 16:54:53 -04:00
util.go
util_test.go
watch_handler.go
watch_handler_test.go removes ioutil usage everywhere which was deprecated in go1.16 (#15297) 2022-11-10 10:26:01 -06:00