Commit Graph

1913 Commits

Author SHA1 Message Date
Daniel Nephin 09d0876b6c command: remove unused logOutput field 2020-03-30 14:11:27 -04:00
Matt Keeler 37694bbb9c
Add optional JSON format to the ACL CLI commands output (#7198)
* Add ACL CLI commands output format option.

Add command level formatter, that incapsulates command output printing
logiс that depends on the command `-format` option.
Move Print* functions from acl_helpers to prettyFormatter. Add jsonFormatter.

* Return error code in case of formatting failure.

* Add acl commands -format option to doc.
2020-03-26 13:16:21 -04:00
Freddy 18d356899c
Enable CLI to register terminating gateways (#7500)
* Enable CLI to register terminating gateways

* Centralize gateway proxy configuration
2020-03-26 10:20:56 -06:00
Artur Mullakhmetov 845b9c23fe Return error code in case of formatting failure. 2020-03-26 19:05:10 +03:00
Artur Mullakhmetov eab5b81d91 Add ACL CLI commands output format option.
Add command level formatter, that incapsulates command output printing
logiс that depends on the command `-format` option.
Move Print* functions from acl_helpers to prettyFormatter. Add jsonFormatter.
2020-03-26 19:05:10 +03:00
Daniel Nephin e5d6273a48
command/envoy: Refactor flag parsing/validation (#7504) 2020-03-26 08:19:21 -06:00
Daniel Nephin a95974cf79 Remove unnecessary methods
They call only a single method and add no additional functionality
2020-03-24 18:35:07 -04:00
Daniel Nephin 8df3746927 cmd: use env vars as defaults
Insted of setting them afterward in Run.

This change required a small re-ordering of the test to patch the
environment before calling New()
2020-03-24 18:34:46 -04:00
Daniel Nephin 6e10616b13 Fix tests failing on master
The default version was changed in https://github.com/hashicorp/consul/pull/7452
which caused these tests to fail.
2020-03-23 16:38:14 -04:00
Hans Hasselberg d5f4b8c3a3
envoy: default to 1.13.1 (#7452) 2020-03-17 22:23:42 +01:00
Hans Hasselberg bf6a91af94
docs: fix filenames (#7453) 2020-03-17 21:00:45 +01:00
Chris Piraino d3dd49b79c
Log "vew version available" message at info level (#7462) 2020-03-17 09:53:15 -05:00
R.B. Boyer dfe5ba134b
fix flaky TestCatalogListNodesCommand_verticalBar test (#7422) 2020-03-10 11:01:13 -05:00
R.B. Boyer 6adad71125
wan federation via mesh gateways (#6884)
This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch:

There are several distinct chunks of code that are affected:

* new flags and config options for the server

* retry join WAN is slightly different

* retry join code is shared to discover primary mesh gateways from secondary datacenters

* because retry join logic runs in the *agent* and the results of that
  operation for primary mesh gateways are needed in the *server* there are
  some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur
  at multiple layers of abstraction just to pass the data down to the right
  layer.

* new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers

* the function signature for RPC dialing picked up a new required field (the
  node name of the destination)

* several new RPCs for manipulating a FederationState object:
  `FederationState:{Apply,Get,List,ListMeshGateways}`

* 3 read-only internal APIs for debugging use to invoke those RPCs from curl

* raft and fsm changes to persist these FederationStates

* replication for FederationStates as they are canonically stored in the
  Primary and replicated to the Secondaries.

* a special derivative of anti-entropy that runs in secondaries to snapshot
  their local mesh gateway `CheckServiceNodes` and sync them into their upstream
  FederationState in the primary (this works in conjunction with the
  replication to distribute addresses for all mesh gateways in all DCs to all
  other DCs)

* a "gateway locator" convenience object to make use of this data to choose
  the addresses of gateways to use for any given RPC or gossip operation to a
  remote DC. This gets data from the "retry join" logic in the agent and also
  directly calls into the FSM.

* RPC (`:8300`) on the server sniffs the first byte of a new connection to
  determine if it's actually doing native TLS. If so it checks the ALPN header
  for protocol determination (just like how the existing system uses the
  type-byte marker).

* 2 new kinds of protocols are exclusively decoded via this native TLS
  mechanism: one for ferrying "packet" operations (udp-like) from the gossip
  layer and one for "stream" operations (tcp-like). The packet operations
  re-use sockets (using length-prefixing) to cut down on TLS re-negotiation
  overhead.

* the server instances specially wrap the `memberlist.NetTransport` when running
  with gateway federation enabled (in a `wanfed.Transport`). The general gist is
  that if it tries to dial a node in the SAME datacenter (deduced by looking
  at the suffix of the node name) there is no change. If dialing a DIFFERENT
  datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh
  gateways to eventually end up in a server's :8300 port.

* a new flag when launching a mesh gateway via `consul connect envoy` to
  indicate that the servers are to be exposed. This sets a special service
  meta when registering the gateway into the catalog.

* `proxycfg/xds` notice this metadata blob to activate additional watches for
  the FederationState objects as well as the location of all of the consul
  servers in that datacenter.

* `xds:` if the extra metadata is in place additional clusters are defined in a
  DC to bulk sink all traffic to another DC's gateways. For the current
  datacenter we listen on a wildcard name (`server.<dc>.consul`) that load
  balances all servers as well as one mini-cluster per node
  (`<node>.server.<dc>.consul`)

* the `consul tls cert create` command got a new flag (`-node`) to help create
  an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 15:59:02 -05:00
Alex Dzyoba 4137d06f9f
command: change delim in columnize to funny node names (#6652)
When node name contains vertical bar symbol some commands output is
garbled because `|` is used as a delimiter in `columnize.SimpleFormat`.

This commit changes format string to use `\x1f` - ASCII unit
separator[1] as a delimiter and also adds test to cover this case.

Affected commands:

* `consul catalog nodes`
* `consul members`
* `consul operator raft list-peers`
* `consul intention get`

Fixes #3951.

[1]: https://en.wikipedia.org/wiki/Delimiter#Solutions
2020-03-09 11:24:56 +01:00
Johannes Scheuermann f8ded993af
agent: log error when agent crashes in an early stage (#7411) 2020-03-09 10:45:21 +01:00
Chris Piraino d7fdb249fc
Fix -mesh-gateway flag help text (#7265) 2020-02-11 14:48:58 -06:00
Lars Lehtonen 26d706eecd
monitor: fix dropped error (#7206) 2020-02-11 10:57:50 +01:00
Hans Hasselberg 6739fe6e83
connect: add validations around intermediate cert ttl (#7213) 2020-02-11 00:05:49 +01:00
Hans Hasselberg 9cb7adb304
add envoy version 1.12.2 and 1.13.0 to the matrix (#7240)
* add 1.12.2

* add envoy 1.13.0

* Introduce -envoy-version to get 1.10.0 passing.

* update old version and fix consul-exec case

* add envoy_version and fix check

* Update Envoy CLI tests to account for the 1.13 compatibility changes.

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2020-02-10 14:53:04 -05:00
R.B. Boyer 36982d5274
cli: fix typo in -namespace help text (#7225) 2020-02-05 14:43:25 -06:00
Paschalis Tsilias a335aa57c5
Expose Envoy's /stats for statsd agents (#7173)
* Expose Envoy /stats for statsd agents; Add testcases

* Remove merge conflict leftover

* Add support for prefix instead of path; Fix docstring to mirror these changes

* Add new config field to docs; Add testcases to check that /stats/prometheus is exposed as well

* Parametrize matchType (prefix or path) and value

* Update website/source/docs/connect/proxies/envoy.md

Co-Authored-By: Paul Banks <banks@banksco.de>

Co-authored-by: Paul Banks <banks@banksco.de>
2020-02-03 17:19:34 +00:00
Michael Hofer 4ab3af0ede
docs: add missing Autopilot -min-quorum documentation (#7192) 2020-02-03 10:59:53 +01:00
Lars Lehtonen 699d1b2acb
cli: check previously ignored errors when updating a policy (#6565) 2020-02-03 10:14:30 +01:00
R.B. Boyer cf29bd4dcf
cli: improve the file safety of 'consul tls' subcommands (#7186)
- also fixing the signature of file.WriteAtomicWithPerms
2020-01-31 10:12:36 -06:00
Matt Keeler bfc03ec587
Fix a couple bugs regarding intentions with namespaces (#7169) 2020-01-29 17:30:38 -05:00
Chris Piraino 401221de58
Allow users to configure either unstructured or JSON logging (#7130)
* hclog Allow users to choose between unstructured and JSON logging
2020-01-28 17:50:41 -06:00
Anthony Scalisi beb928f8de fix spelling errors (#7135) 2020-01-27 07:00:33 -06:00
Matt Keeler c09693e545
Updates to Config Entries and Connect for Namespaces (#7116) 2020-01-24 10:04:58 -05:00
Paul Lesiak 527daba03c bug: Consul lock does not receive signals if lock not acquired (#5909) 2020-01-22 12:44:48 -07:00
Hans Hasselberg 804eb17094
connect: check if intermediate cert needs to be renewed. (#6835)
Currently when using the built-in CA provider for Connect, root certificates are valid for 10 years, however secondary DCs get intermediates that are valid for only 1 year. There is no mechanism currently short of rotating the root in the primary that will cause the secondary DCs to renew their intermediates.
This PR adds a check that renews the cert if it is half way through its validity period.

In order to be able to test these changes, a new configuration option was added: IntermediateCertTTL which is set extremely low in the tests.
2020-01-17 23:27:13 +01:00
Aestek ba8fd8296f Add support for dual stack IPv4/IPv6 network (#6640)
* Use consts for well known tagged adress keys

* Add ipv4 and ipv6 tagged addresses for node lan and wan

* Add ipv4 and ipv6 tagged addresses for service lan and wan

* Use IPv4 and IPv6 address in DNS
2020-01-17 09:54:17 -05:00
Hans Hasselberg ab2f2484ad
connect: use inline_string instead for envoy ca(#7024) 2020-01-10 15:57:54 +01:00
Matt Keeler c276e2a634
Revert "Remove docs refs to NS inference from ACL token" (#6976)
This reverts commit 3a8426de9c.

# Conflicts:
#	command/flags/http.go
#	website/source/api/acl/binding-rules.html.md
#	website/source/api/acl/policies.html.md
#	website/source/api/acl/roles.html.md
#	website/source/api/acl/tokens.html.md
#	website/source/api/kv.html.md
#	website/source/api/session.html.md
#	website/source/docs/commands/_http_api_namespace_options.html.md
2019-12-20 11:52:50 -05:00
Matt Keeler 34e6e23d43
Change how namespaces are specified for the CLI (#6960) 2019-12-18 11:06:39 -05:00
Hans Hasselberg 1661041046
envoy: replace ca filename with inline_bytes. (#6822) 2019-12-13 17:44:48 +01:00
freddygv 3a8426de9c Remove docs refs to NS inference from ACL token 2019-12-10 13:50:28 -07:00
Matt Keeler 5934f803bf
Sync of OSS changes to support namespaces (#6909) 2019-12-09 21:26:41 -05:00
Matt Keeler a704ebe639
Add Namespace support to the API module and the CLI commands (#6874)
Also update the Docs and fixup the HTTP API to return proper errors when someone attempts to use Namespaces with an OSS agent.

Add Namespace HTTP API docs

Make all API endpoints disallow unknown fields
2019-12-06 11:14:56 -05:00
Aestek 6817676223 Fix consul connect token env variable doc (#5942)
The cli documentation for consul connect commands incorrectly indicated
to use CONSUL_TOKEN instead of CONSUL_HTTP_TOKEN env var.
2019-12-04 14:01:03 -06:00
Sarah Adams aed5cb7669
give feedback to CLI user on forceleave command if node does not exist (#6841) 2019-12-02 11:06:15 -08:00
Matt Keeler b069d6777b
OSS KV Modifications to Support Namespaces 2019-11-25 12:57:35 -05:00
Chris Piraino 646da639ee Allow services register command to register an unnamed check
The logic in parsing data files and converting them to data structures
accidentally removed healthchecks with no Name field, even though we
explicitly state in API documentation that is allowed.

We remove the check for "len(results.Checks) == 1" because if the length
of the array is more than 0, we know that it is not a zero value array.
This allows us to register a singular, unnamed check via the CLI.

Fixes #6796
2019-11-15 14:28:36 -06:00
Sarah Christoff 5e1c6e907b
Set MinQuorum variable in Autopilot (#6654)
* Add MinQuorum to Autopilot
2019-10-29 09:04:41 -05:00
rerorero 86c8e48dd9 fix: incorrect struct tag and WaitGroup usage (#6649)
* remove duplicated json tag

* fix: incorrect wait group usage
2019-10-18 13:59:29 -04:00
Sarah Christoff 5e26971864
Prune Unhealthy Agents (#6571)
* Add -prune flag to ForceLeave
2019-10-04 16:10:02 -05:00
Freddy fdd10dd8b8
Expose HTTP-based paths through Connect proxy (#6446)
Fixes: #5396

This PR adds a proxy configuration stanza called expose. These flags register
listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only
listening on the loopback interface, while still accepting traffic from non
Connect-enabled services.

Under expose there is a boolean checks flag that would automatically expose all
registered HTTP and gRPC check paths.

This stanza also accepts a paths list to expose individual paths. The primary
use case for this functionality would be to expose paths for third parties like
Prometheus or the kubelet.

Listeners for requests to exposed paths are be configured dynamically at run
time. Any time a proxy, or check can be registered, a listener can also be
created.

In this initial implementation requests to these paths are not
authenticated/encrypted.
2019-09-25 20:55:52 -06:00
Sarah Adams 001137e5e5
test: ensure all TestAgent constructions use a constructor (#6443)
ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place

Fixes #6435
2019-09-05 10:24:36 -07:00
Sarah Adams 4ed5515fca
refactor & add better retry logic to NewTestAgent (#6363)
Fixes #6361
2019-09-03 15:05:51 -07:00
R.B. Boyer ae79cdab1b
connect: introduce ExternalSNI field on service-defaults (#6324)
Compiling this will set an optional SNI field on each DiscoveryTarget.
When set this value should be used for TLS connections to the instances
of the target. If not set the default should be used.

Setting ExternalSNI will disable mesh gateway use for that target. It also 
disables several service-resolver features that do not make sense for an 
external service.
2019-08-19 12:19:44 -05:00