Commit Graph

204 Commits

Author SHA1 Message Date
R.B. Boyer c6c4a2251a Merge Consul OSS branch master at commit b3541c4f34 2019-07-26 10:34:24 -05:00
Alvin Huang 3558f9cf6d
add dev docker image upload (#5879)
* add dev docker image upload

* remove the go cache since that isn't needed

* add comment and image labels

* get password from stdin
2019-07-25 09:19:09 -04:00
Matt Keeler 52c608df78
`make test-docker` (#6059)
* Implement the test-docker make target

Running tests within docker allows us to resource constrain them better to not take over our systems. Additionally it allows us to run the tests on linux instead of the host OS which often times is macOS.

* Use GOMAXPROCS instead of -p

* Add a comment about docker cpus
2019-07-04 10:22:59 -04:00
Paul Banks 421ecd32fc
Connect: allow configuring Envoy for L7 Observability (#5558)
* Add support for HTTP proxy listeners

* Add customizable bootstrap configuration options

* Debug logging for xDS AuthZ

* Add Envoy Integration test suite with basic test coverage

* Add envoy command tests to cover new cases

* Add tracing integration test

* Add gRPC support WIP

* Merged changes from master Docker. get CI integration to work with same Dockerfile now

* Make docker build optional for integration

* Enable integration tests again!

* http2 and grpc integration tests and fixes

* Fix up command config tests

* Store all container logs as artifacts in circle on fail

* Add retries to outer part of stats measurements as we keep missing them in CI

* Only dump logs on failing cases

* Fix typos from code review

* Review tidying and make tests pass again

* Add debug logs to exec test.

* Fix legit test failure caused by upstream rename in envoy config

* Attempt to reduce cases of bad TLS handshake in CI integration tests

* bring up the right service

* Add prometheus integration test

* Add test for denied AuthZ both HTTP and TCP

* Try ANSI term for Circle
2019-04-29 17:27:57 +01:00
Matt Keeler 222afeae4c
Move the watch package into the api module (#5664)
* Move the watch package into the api module

It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests.

The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)
2019-04-26 12:33:01 -04:00
Matt Keeler c6be3b525b
Build System Fixes for Go Modules (#5655)
* Docker based builds can now use the module cache

* Simplify building the consul-dev docker image.

* Make sure to pull the latest consul image.

* Allow selecting base image version for the dev image
2019-04-12 15:17:13 -04:00
Freddy b2c12550d7
Remove old UI, option to use it, and its build processes 2019-04-12 09:02:27 -06:00
Alvin Huang ffa123b73e download tools without library 2019-04-10 13:09:02 -04:00
Paul Banks 11362e77be
build: use only version tags in version output now api is tagged too (#5622)
* build: use only version tags in version output now api is tagged too

Fixes #5621

Since we now have api package tags, our build tooling was picking up api tag when working out version to bake into builds.

This fixes it by restricting to only tags that start with `v`.

Before:

```
$ make version
Version:                    1.4.4
Version + release:          1.4.4-dev
Version + git:              api/v1.0.1-90-g3ce60db0c
Version + release + git:    api/v1.0.1-90-g3ce60db0c-dev (3ce60db0c)
```

After:

```
$ make version
Version:                    1.4.4
Version + release:          1.4.4-dev
Version + git:              v1.4.4-126-g3ce60db0c
Version + release + git:    v1.4.4-126-g3ce60db0c-dev (3ce60db0c)
```

* Update GNUmakefile
2019-04-10 12:54:03 +01:00
Alvin Huang 8e07c63c54
removing gocov packages (#5534)
* removing gocov packages

* revise cov make target
2019-04-09 16:05:52 -04:00
R.B. Boyer 27446a19e2
remove remaining references to govendor and vendorfmt (#5587) 2019-04-01 09:55:48 -05:00
Hans Hasselberg ac45b17482
fix remaining CI failures after Go 1.12.1 Upgrade (#5576) 2019-03-29 16:29:27 +01:00
Mitchell Hashimoto f76022fa63 CA Provider Plugins (#4751)
This adds the `agent/connect/ca/plugin` library for consuming/serving Connect CA providers as [go-plugin](https://github.com/hashicorp/go-plugin) plugins. This **does not** wire this up in any way to Consul itself, so this will not enable using these plugins yet. 

## Why?

We want to enable CA providers to be pluggable without modifying Consul so that any CA or PKI system can potentially back the Connect certificates. This CA system may also be used in the future for easier bootstrapping and internal cluster security.

### go-plugin

The benefit of `go-plugin` is that for the plugin consumer, the fact that the interface implementation is communicating over multi-process RPC is invisible. Internals of Consul will continue to just use `ca.Provider` interface implementations as if they're local. For plugin _authors_, they simply have to implement the interface. The network/transport/process management issues are handled by go-plugin itself.

The CA provider plugins support both `net/rpc` and gRPC transports. This enables easy authoring in any language. go-plugin handles the actual protocol handshake and connection. This is just a feature of go-plugin. 

`go-plugin` is already in production use for years by Packer, Terraform, Nomad, Vault, and Sentinel. We've shown stability for both desktop and server-side software. It is very mature.

## Implementation Details

### `map[string]interface{}`

The `Configure` method passes a `map[string]interface{}`. This map contains only Go primitives and containers of primitives (no funcs, chans, etc.). For `net/rpc` we encode as-is using Gob. For gRPC we marshal to JSON and transmit as a `bytes` type. This is the same approach we take with Vault and other software.

Note that this is just the transport protocol, the end software views it fully decoded.

### `x509.Certificate` and `CertificateRequest`

We transmit the raw ASN.1  bytes and decode on the other side. Unit tests are verifying we get the same cert/csrs across the wire.

### Testing

`go-plugin` exposes test helpers that enable testing the full plugin RPC over real loopback network connections. We test all endpoints for success and error for both `net/rpc` and gRPC.

### Vendoring

This PR doesn't introduce vendoring for two reasons:

  1. @banks's `f-envoy` branch introduces a lot of these and I didn't want conflict.
  2. The library isn't actually used yet so it doesn't introduce compile-time errors (it does introduce test errors).

## Next Steps

With this in place, we need to figure out the proper way to actually hook these up to Consul, load them, etc. This discussion can happen elsewhere, since regardless of approach this plugin library implementation is the exact same.
2019-01-07 12:48:44 -05:00
Matt Keeler 9819683653
Single quote a directory (#4846)
Allows building the dev docker container to work when you have spaces in your cwd.
2018-10-24 09:48:19 -04:00
Matt Keeler 18b29c45c4
New ACLs (#4791)
This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week.
Description

At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers.

On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though.

    Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though.
    All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management.
    Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are:
        A server running the new system must still support other clients using the legacy system.
        A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system.
        The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode.

So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 12:04:07 -04:00
Freddy 524192c919 Improve resilience of api pkg tests (#4676)
* Add function to wait for serfHealth in api tests

* Disable connect when creating semaphore test clients

* Wait for serfHealth when creating sessions in their tests

* Add helper functions to create lock/semaphore sessions without checks

* Log passing tests to prevent timeout in Travis due to lack of output
2018-09-18 17:47:01 +01:00
Freddy b2032b05be
Add script and makefile goal to help debug flaky tests 2018-09-10 16:44:07 +01:00
Pierre Souchay 80b6653a8e Keep same parameters on retry so results can be cached by go test (#4627) 2018-09-04 12:27:39 +01:00
Pierre Souchay cec5d72396 BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472)
- Improve resilience of testrpc.WaitForLeader()

- Add additionall retry to CI

- Increase "go test" timeout to 8m

- Add wait for cluster leader to several tests in the agent package

- Add retry to some tests in the api and command packages
2018-08-06 19:46:09 -04:00
Paul Banks 16200e4c75
Refactor test retry to only affect CI (#4436)
* Refactor test retry to only affect CI

* Move test install deps out of the retry loop

* Add internal targets to PHONY too
2018-07-24 15:12:48 +01:00
Pierre Souchay 68f7f85cda Tune GNUMakefile to pass more easily Travis tests 2018-07-10 22:55:37 +02:00
Pierre Souchay 817a69edad Avoid travis to fail with flacky tests 2018-07-10 22:55:37 +02:00
Pierre Souchay 684674b757 Trying to fix Travis tests 2018-07-02 00:24:37 +02:00
Matt Keeler 2cc7cd32af Fix default make target to build everything 2018-06-27 14:25:49 -04:00
Matt Keeler b2989f82d6 Default dev-build to not using GOX 2018-06-26 14:26:23 -04:00
Matt Keeler c3c763a52e Fix bug in makefile that prevented running the website publish. 2018-06-26 14:08:10 -04:00
Matt Keeler 8f2a19a747 Update verify.sh script 2018-06-26 12:08:33 -04:00
Matt Keeler 5ff7a65e7f Added capability to make dev-tree without pushing
No push is the default
2018-06-26 11:46:37 -04:00
mkeeler 6813a99081 Merge remote-tracking branch 'connect/f-connect' 2018-06-25 19:42:51 +00:00
Matt Keeler 90e74da6b7 Fix a couple find warnings on linux
Additionally add the ability to use go install for dev builds rather than gox (travis doesn’t have gox)
2018-06-19 10:49:07 -04:00
Matt Keeler 12d14f6c54 Allow for building pre-releases/rcs/betas 2018-06-18 17:06:38 -04:00
Matt Keeler 075f4b64ae Quote some make vars 2018-06-18 14:39:34 -04:00
Matt Keeler 3f290cc676 Generalize git pushing in a bash function 2018-06-15 20:42:07 -04:00
Matt Keeler 147e4a1eae Add capability to put tree back into dev mode via make dev-tree 2018-06-15 08:00:12 -04:00
Matt Keeler 48910f7583 Update the scripting
Automated putting the source tree into release mode.
2018-06-14 21:42:47 -04:00
Mitchell Hashimoto 6031c58ef9
Remove temporary hacks from Makefile 2018-06-14 09:42:10 -07:00
Mitchell Hashimoto ecb05cc957
Add Makefile hack for tests to run 2018-06-14 09:42:02 -07:00
Paul Banks 36dbd878c9
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list. 2018-06-14 09:41:58 -07:00
Paul Banks 662e57d91b
Make test output more useful now we uses testify with multi-line error messages 2018-06-14 09:41:58 -07:00
Paul Banks 8d09381b96
Super ugly hack to get TeamCity build to work for this PR without adding a vendor that is being added elsewhere and will conflict... 2018-06-14 09:41:58 -07:00
Matt Keeler 56a7d15fbc Add more functionality related to verifying a build and publishing 2018-06-13 17:03:18 -04:00
Matt Keeler c90d7f5160 Redo the build system
Improvements:
   - More modular
   - Building within docker doesn’t use volumes so can be run on a remote docker host
   - Build containers include only minimal context so they only rarely need to be rebuilt and most of the time can be used from the cache.
   - 3 build containers instead of 1. One based off of the upstream golang containers for building go stuff with all our required GOTOOLS installed. One like the old container based off ubuntu bionic for building the old UI (didn’t bother creating a much better container as this shouldn’t be needed once we completely remove the legacy UI). One for building the new UI. Its alpine based with all the node, ember, yarn stuff installed.
   - Top level makefile has the ability to do a container based build without running make dist
   - Can build for arbitrary platforms at the top level using: make consul-docker XC_OS=… XC_ARCH=…
   - overridable functionality to allow for customizations to the enterprise build (like to generate multiple binaries)
   - unified how we compile our go. always use gox even for dev-builds or rather always use the tooling around our scripts which will make sure things get copied to the correct places throughout the filesystem.
2018-06-12 16:55:52 -04:00
Matt Keeler f5a22f8490 Initial progress on build system updates 2018-06-08 10:20:54 -04:00
John Cowen e5eeb0aa7c
UI V2 (#4086)
* Move settings to use the same service/route API as the rest of the app

* Put some ideas down for unit testing on adapters

* Favour `Model` over `Entity`

* Move away from using `reopen` to using Mixins

* Amend messages, comment/document some usage

* Make sure the returns are consistent in normalizePayload, also

Add some todo's in to remind me to think consider this further at a
later date. For example, is normalizePayload to be a hook or an
overridable method

* Start stripping back the HTML to semantics

* Use a variable rather than chaining

* Remove unused helpers

* Start picking through the new designs, start with listing pages

* First draft HTML for every page

* Making progress on the CSS

* Keep plugging away at the catalog css

* Looking at scrolling

* Wire up filtering

* Sort out filter counting, more or less done a few outstanding

* Start knocking the forms into shape

* Add in codemirror

* Keep moving forwards with the form like layouts

* Start looking at ACL editing page, add footer in

* Pull the filters back in, look at an autoresizer for scroll views

* First draft toggles

* 2nd draft healthcheck icons

* Tweak node healthcheck icons

* Looking at healthcheck detail icons

* Tweak the filter-bar and add selections to the in content tabs

* Add ACL create, pill-like acl type highlight

* Tweaking the main nav some more

* Working on the filter-bar and freetext-filter

* Masonry layout

* Stick with `checks` instead of healthy/unhealthy

* Fix up the filter numbers/counts

* Use the thead for a measure

* First draft tomography back in

* First draft DC dropdown

* Add a temporary create buttong to kv's

* Move KV and ACL to use a create page

* Move tags

* Run through old tests

* Injectable server

* Start adding test attributes

* Add some page objects

* More test attributes and pages

* Acl filter objects

* Add a page.. page object

* Clickable items in lists

* Add rest/spread babel plugin, remove mirage for now

* Add fix for ember-collection

* Keep track of acl filters

* ember-cli-page-object

* ember-test-selectors

* ui: update version of ui compile deps

* Update static assets

* Centralize radiogroup helper

* Rejig KV's and begin to clean it up

* Work around lack of Tags for the moment..

* Some little css tweaks and start to remove possibles

* Working on the dc page and incidentals

1. Sort the datacenter-picker list
2. Add a selected state to the datacenter-picker
3. Make dc an {Name: dc}
4. Add an env helper to get to 'env vars' from within templates

* Click outside stuff for the datacenter-picker, is-active on nav

* Make sure the dropdown CTA can be active

* Bump ember add pluralize helper

* Little try at sass based custom queries

* Rejig tablular collection so it deals with resizing, actions

1. WIP: start building actions dropdowns
2. Move tabular collection to deal with resizing to rule out differences

* First draft actions dropdowns

* Add ports, selectable IP's

* Flash messages, plus general cleanup/consistency

1. Add ember-cli-flash for flash messages
2. Move everything to get() instead of item.get
3. Spotted a few things that weren't consistent

* DOn't go lower than zero

* First draft vertical menu

* Missed a get, tweak dropmenu tick

* Big cleanup

1. this.get(), this.set() > get(), set()
2. assign > {...{}, ...{}}
3. Seperator > separator

* WIP: settings

* Moved things into a ui-v2 folder

* Decide on a way to do the settings page whilst maintaining the url + dc's

* Start some error pages

* Remove base64 polyfill

* Tie in settings, fix atob bug, tweak layout css

* Centralize confirmations into a component

* Allow switching between the old and new UI with the CONSUL_UI_BETA env var

Currently all the assets are packaged into a single AssetFS and a prefix is configured to switch between the two.

* Attempt at some updates to integrate the v2 ui build into the main infrastructure

* Add redirect to index.html for unknown paths

* Allow redictor to /index.html for new ui when using -ui-dir

* Take ACLs to the correct place on save

* First pass breadcrumbs

* Remove datacenter selector on the index page

* Tweak overall layout

* Make buttons 'resets'

* Tweak last DC stuff

* Validations plus kv keyname viewing tweaks

* Pull sessions back in

* Tweak the env vars to be more reusable

* Move isAnon to the view

* No items and disabled acl css

* ACL and KV details

1. Unauthorized page
2. Make sure the ACL is always selected when it needs it
3. Check record deletion with a changeset

* Few more acl tweaks/corrections

* Add no items view to node > services

* Tags for node > services

* Make sure we have tags

* Fix up the labels on the tomography graph

* Add node link (agent) to kv sessions

* Duplicate up `create` for KV 'root creation'

* Safety check for health checks

* Fix up the grids

* Truncate td a's, fix kv columns

* Watch for spaces in KV id's

* Move actions to their own mixins for now at least

* Link reset to settings incase I want to type it in

* Tweak error page

* Cleanup healthcheck icons in service listing

* Centralize errors and make getting back easier

* Nice numbers

* Compact buttons

* Some incidental css cleanups

* Use 'Key / Value' for root

* Tweak tomography layout

* Fix single healthcheck unhealthy resource

* Get loading screen ready

* Fix healthy healthcheck tick

* Everything in header starts white

* First draft loader

* Refactor the entire backend to use proper unique keys, plus..

1. Make unique keys form dc + slug (uid)
2. Fun with errors...

* Tweak header colors

* Add noopener noreferrer to external links

* Add supers to setupController

* Implement cloning, using ember-data...

* Move the more expensive down the switch order

* First draft empty record cleanup..

* Add the cusomt store test

* Temporarily use the htmlSafe prototype to remove the console warning

* Encode hashes in urls

* Go back to using title for errors for now

* Start removing unused bulma

* Lint

* WIP: Start looking at failing tests

* Remove single redirect test

* Finish off error message styling

* Add full ember-data cache invalidation to avoid stale data...

* Add uncolorable warning icons

* More info icon

* Rearrange single service, plus tag printing

* Logo

* No quotes

* Add a simple startup logo

* Tweak healthcheck statuses

* Fix border-color for healthchecks

* Tweak node tabs

* Catch 401 ACL errors and rethrow with the provided error message

* Remove old acl unauth and error routes

* Missed a super

* Make 'All' refer to number of checks, not services

* Remove ember-resizer, add autoprefixer

* Don't show tomography if its not worth it, viewify it more also

* Little model cleanup

* Chevrons

* Find a way to reliably set the class of html from the view

* Consistent html

* Make sure session id's are visible as long as possible

* Fix single service check count

* Add filters and searchs to the query string

* Don't remember the selected tab

* Change text

* Eror tweaking

* Use chevrons on all breadcrumbs even in kv's

* Clean up a file

* Tweak some messaging

* Makesure the footer overlays whats in the page

* Tweak KV errors

* Move json toggle over to the right

* feedback-dialog along with copy buttons

* Better confirmation dialogs

* Add git sha comment

* Same title as old UI

* Allow defaults

* Make sure value is a string

* WIP: Scrolling dropdowns/confirmations

* Add to kv's

* Remove set

* First pass trace

* Better table rows

* Pull over the hashi code editor styles

* Editor tweaks

* Responsive tabs

* Add number formatting to tomography

* Review whats left todo

* Lint

* Add a coordinate ember data triplet

* Bump in a v2.0.0

* Update old tests

* Get coverage working again

* Make sure query keys are also encoded

* Don't test console.error

* Unit test some more utils

* Tweak the size of the tabular collections

* Clean up gitignore

* Fix copy button rollovers

* Get healthcheck 'icon icons' onto the text baseline

* Tweak healthcheck padding and alignment

* Make sure commas kick in in rtt, probably never get to that

* Improve vertical menu

* Tweak dropdown active state to not have a bg

* Tweak paddings

* Search entire string not just 'startsWith'

* Button states

* Most buttons have 1px border

* More button tweaks

* You can only view kv folders

* CSS cleanup reduction

* Form input states and little cleanup

* More CSS reduction

* Sort checks by importance

* Fix click outside on datacenter picker

* Make sure table th's also auto calculate properly

* Make sure `json` isn't remembered in KV editing

* Fix recursive deletion in KV's

* Centralize size

* Catch updateRecord

* Don't double envode

* model > item consistency

* Action loading and ACL tweaks

* Add settings dependencies to acl tests

* Better loading

* utf-8 base64 encode/decode

* Don't hang off a prototype for htmlSafe

* Missing base64 files...

* Get atob/btoa polyfill right

* Shadowy rollovers

* Disabled button styling for primaries

* autofocuses only onload for now

* Fix footer centering

* Beginning of 'notices'

* Remove the isLocked disabling as we are letting you do what the API does

* Don't forget the documentation link for sessions

* Updates are more likely

* Use exported constant

* Dont export redirectFS and a few other PR updates

* Remove the old bootstrap config which was used for the old UI skin

* Use curlies for multiple properties
2018-05-10 19:52:53 +01:00
Kyle Havlovitz af4be34a2a
Update make static-assets goal and run format 2018-04-13 09:57:25 -07:00
Paul Banks 9b97bebc95
Merge pull request #3903 from hashicorp/build-fixes
[WIP] Attempt to find some low-hanging fruit for CI failures
2018-02-23 13:12:45 +00:00
Kyle Havlovitz 1e11082709
Use GOTAGS in the vet make goal 2018-02-22 15:57:09 -08:00
Paul Banks 7b85f09d59
Travis evaluates ENV before cloning git repo and cding so we need to delay gathering packages until the makefile 2018-02-21 12:54:23 +00:00
Paul Banks 41ae19c677
Split the heavy test packages out to their own Jobs. 2018-02-21 12:20:33 +00:00
Paul Banks 29feb66fbd
Portability! 2018-02-20 15:48:18 +00:00
Paul Banks 6f266a0103
Only output sparse lines to keep Travis happy while logging verbosely to disk 2018-02-20 15:33:36 +00:00
Paul Banks e416f828dc
Output from test as we run to stop travis killing us and try no parallelism to avoid timing issues on their contended CPUs 2018-02-20 13:16:53 +00:00
James Phillips fbfe0c65a7
Switches to HashiCorp fork of jteeuwen/go-bindata.
This was reported in #3868. We make a HashiCorp hard fork of the
jteeuwen/go-bindata hard fork that was replaced and diffed the code
against a Dec 1, 2015 copy of the original repository we had as a
cross-check of that hard fork.

This replaces references to jteeuwen/go-bindata to point to the
HashiCorp fork.

Fixes #3868
2018-02-07 07:39:10 -08:00
Chad Whitacre d11a664e25 Allow for default GOPATH 2018-01-04 14:38:20 -05:00
James Phillips 5d574e7fc7
Gets rid of full log in Travis for now. 2017-11-08 19:39:02 -08:00
Frank Schroeder c94751ad43 test: replace porter tool with freeport lib
This patch removes the porter tool which hands out free ports from a
given range with a library which does the same thing. The challenge for
acquiring free ports in concurrent go test runs is that go packages are
tested concurrently and run in separate processes. There has to be some
inter-process synchronization in preventing processes allocating the
same ports.

freeport allocates blocks of ports from a range expected to be not in
heavy use and implements a system-wide mutex by binding to the first
port of that block for the lifetime of the application. Ports are then
provided sequentially from that block and are tested on localhost before
being returned as available.
2017-10-21 22:01:09 +02:00
Frank Schroeder a61929a597 build: add changelogfmt target to format [GH-xxxx] references to links 2017-10-04 21:14:16 +02:00
Frank Schröder 12216583a1 New config parser, HCL support, multiple bind addrs (#3480)
* new config parser for agent

This patch implements a new config parser for the consul agent which
makes the following changes to the previous implementation:

 * add HCL support
 * all configuration fragments in tests and for default config are
   expressed as HCL fragments
 * HCL fragments can be provided on the command line so that they
   can eventually replace the command line flags.
 * HCL/JSON fragments are parsed into a temporary Config structure
   which can be merged using reflection (all values are pointers).
   The existing merge logic of overwrite for values and append
   for slices has been preserved.
 * A single builder process generates a typed runtime configuration
   for the agent.

The new implementation is more strict and fails in the builder process
if no valid runtime configuration can be generated. Therefore,
additional validations in other parts of the code should be removed.

The builder also pre-computes all required network addresses so that no
address/port magic should be required where the configuration is used
and should therefore be removed.

* Upgrade github.com/hashicorp/hcl to support int64

* improve error messages

* fix directory permission test

* Fix rtt test

* Fix ForceLeave test

* Skip performance test for now until we know what to do

* Update github.com/hashicorp/memberlist to update log prefix

* Make memberlist use the default logger

* improve config error handling

* do not fail on non-existing data-dir

* experiment with non-uniform timeouts to get a handle on stalled leader elections

* Run tests for packages separately to eliminate the spurious port conflicts

* refactor private address detection and unify approach for ipv4 and ipv6.

Fixes #2825

* do not allow unix sockets for DNS

* improve bind and advertise addr error handling

* go through builder using test coverage

* minimal update to the docs

* more coverage tests fixed

* more tests

* fix makefile

* cleanup

* fix port conflicts with external port server 'porter'

* stop test server on error

* do not run api test that change global ENV concurrently with the other tests

* Run remaining api tests concurrently

* no need for retry with the port number service

* monkey patch race condition in go-sockaddr until we understand why that fails

* monkey patch hcl decoder race condidtion until we understand why that fails

* monkey patch spurious errors in strings.EqualFold from here

* add test for hcl decoder race condition. Run with go test -parallel 128

* Increase timeout again

* cleanup

* don't log port allocations by default

* use base command arg parsing to format help output properly

* handle -dc deprecation case in Build

* switch autopilot.max_trailing_logs to int

* remove duplicate test case

* remove unused methods

* remove comments about flag/config value inconsistencies

* switch got and want around since the error message was misleading.

* Removes a stray debug log.

* Removes a stray newline in imports.

* Fixes TestACL_Version8.

* Runs go fmt.

* Adds a default case for unknown address types.

* Reoders and reformats some imports.

* Adds some comments and fixes typos.

* Reorders imports.

* add unix socket support for dns later

* drop all deprecated flags and arguments

* fix wrong field name

* remove stray node-id file

* drop unnecessary patch section in test

* drop duplicate test

* add test for LeaveOnTerm and SkipLeaveOnInt in client mode

* drop "bla" and add clarifying comment for the test

* split up tests to support enterprise/non-enterprise tests

* drop raft multiplier and derive values during build phase

* sanitize runtime config reflectively and add test

* detect invalid config fields

* fix tests with invalid config fields

* use different values for wan sanitiziation test

* drop recursor in favor of recursors

* allow dns_config.udp_answer_limit to be zero

* make sure tests run on machines with multiple ips

* Fix failing tests in a few more places by providing a bind address in the test

* Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder.

* Add porter to server_test.go to make tests there less flaky

* go fmt
2017-09-25 11:40:42 -07:00
James Phillips 3b1eebd502 Gets rid of spurious $(GOTEST_FLAGS) in the log file name. 2017-09-07 12:32:58 -07:00
James Phillips b1a15e0c3d
Adds open source side of network segments (feature is Enterprise-only). 2017-08-30 11:58:29 -07:00
Frank Schroeder f25632d6a1 build: drop 'consul' build tag for OSS build 2017-08-30 13:40:18 +02:00
James Phillips bddca46d0e
Revert "Rename GNUMakefile to Makefile (#3348)"
This reverts commit b0adb8af92.
2017-08-03 09:50:04 -07:00
Seth Vargo b0adb8af92 Rename GNUMakefile to Makefile (#3348) 2017-08-02 10:46:03 -07:00
Frank Schroeder 2959b16baa
build: fix travis build 2017-07-18 09:24:40 +02:00
Frank Schroeder ad9029b017 build: use vendorfmt to make vendor.json more merge friendly 2017-07-18 09:15:25 +02:00
Frank Schroeder 9e83a034e3
build: run vet on make test 2017-07-11 16:16:57 -07:00
Frank Schroeder 61ca9b02d4 build: build all packages together
Build all packages together with a global timeout. Locally,
the tests pass within 40 sec. On travis, we complete within
3-4 min.

travis truncates the logs after 4MB which we are hitting so
show the relevant information first and then whatever is left.
2017-07-07 09:22:34 +02:00
Maciej Stępyra b775a85624 Missing tools in makefile (#3205) 2017-06-29 12:55:01 +02:00
Preetha Appan d5d79d0ad3 fix type in make file 2017-06-28 09:48:00 -05:00
preetapan 99f93cd4e7 make test should fail fast on test compilation errors (#3172) 2017-06-21 14:47:08 -07:00
James Phillips 0e695aef94 Fixes checked in web assets and associated build scripts. (#3173) 2017-06-21 14:43:07 -07:00
Preetha Appan cf5006c4f4 Fixed static asset build target and checked in new executable for assetfs 2017-06-12 12:57:02 -05:00
Frank Schroeder 1c75cf1af5 pkg refactor
command/agent/*                  -> agent/*
    command/consul/*                 -> agent/consul/*
    command/agent/command{,_test}.go -> command/agent{,_test}.go
    command/base/command.go          -> command/base.go
    command/base/*                   -> command/*
    commands.go                      -> command/commands.go

The script which did the refactor is:

(
	cd $GOPATH/src/github.com/hashicorp/consul
	git mv command/agent/command.go command/agent.go
	git mv command/agent/command_test.go command/agent_test.go
	git mv command/agent/flag_slice_value{,_test}.go command/
	git mv command/agent .
	git mv command/base/command.go command/base.go
	git mv command/base/config_util{,_test}.go command/
	git mv commands.go command/
	git mv consul agent
	rmdir command/base/

	gsed -i -e 's|package agent|package command|' command/agent{,_test}.go
	gsed -i -e 's|package agent|package command|' command/flag_slice_value{,_test}.go
	gsed -i -e 's|package base|package command|' command/base.go command/config_util{,_test}.go
	gsed -i -e 's|package main|package command|' command/commands.go

	gsed -i -e 's|base.Command|BaseCommand|' command/commands.go
	gsed -i -e 's|agent.Command|AgentCommand|' command/commands.go
	gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/commands.go
	gsed -i -e 's|base\.||' command/commands.go
	gsed -i -e 's|command\.||' command/commands.go

	gsed -i -e 's|command|c|' main.go
	gsed -i -e 's|range Commands|range command.Commands|' main.go
	gsed -i -e 's|Commands: Commands|Commands: command.Commands|' main.go

	gsed -i -e 's|base\.BoolValue|BoolValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.DurationValue|DurationValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.StringValue|StringValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.UintValue|UintValue|' command/operator_autopilot_set.go

	gsed -i -e 's|\bCommand\b|BaseCommand|' command/base.go
	gsed -i -e 's|BaseCommand Options|Command Options|' command/base.go
	gsed -i -e 's|base.Command|BaseCommand|' command/*.go
	gsed -i -e 's|c\.Command|c.BaseCommand|g' command/*.go
	gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/*_test.go
	gsed -i -e 's|base\.||' command/*_test.go

	gsed -i -e 's|\bCommand\b|AgentCommand|' command/agent{,_test}.go
	gsed -i -e 's|cmd.AgentCommand|cmd.BaseCommand|' command/agent.go

	gsed -i -e 's|cli.AgentCommand = new(Command)|cli.Command = new(AgentCommand)|' command/agent_test.go
	gsed -i -e 's|exec.AgentCommand|exec.Command|' command/agent_test.go
	gsed -i -e 's|exec.BaseCommand|exec.Command|' command/agent_test.go
	gsed -i -e 's|NewTestAgent|agent.NewTestAgent|' command/agent_test.go
	gsed -i -e 's|= TestConfig|= agent.TestConfig|' command/agent_test.go
	gsed -i -e 's|: RetryJoin|: agent.RetryJoin|' command/agent_test.go

	gsed -i -e 's|\.\./\.\./|../|' command/config_util_test.go

	gsed -i -e 's|\bverifyUniqueListeners|VerifyUniqueListeners|' agent/config{,_test}.go command/agent.go
	gsed -i -e 's|\bserfLANKeyring\b|SerfLANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
	gsed -i -e 's|\bserfWANKeyring\b|SerfWANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
	gsed -i -e 's|\bNewAgent\b|agent.New|g' command/agent{,_test}.go
	gsed -i -e 's|\bNewAgent|New|' agent/{acl_test,agent,testagent}.go

	gsed -i -e 's|\bAgent\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bBool\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bDefaultConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bDevConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bMergeConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bReadConfigPaths\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bParseMetaPair\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bSerfLANKeyring\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bSerfWANKeyring\b|agent.&|g' command/agent{,_test}.go

	gsed -i -e 's|circonus\.agent|circonus|g' command/agent{,_test}.go
	gsed -i -e 's|logger\.agent|logger|g' command/agent{,_test}.go
	gsed -i -e 's|metrics\.agent|metrics|g' command/agent{,_test}.go
	gsed -i -e 's|// agent.Agent|// agent|' command/agent{,_test}.go
	gsed -i -e 's|a\.agent\.Config|a.Config|' command/agent{,_test}.go

	gsed -i -e 's|agent\.AppendSliceValue|AppendSliceValue|' command/{configtest,validate}.go

	gsed -i -e 's|consul/consul|agent/consul|' GNUmakefile

	gsed -i -e 's|\.\./test|../../test|' agent/consul/server_test.go

	# fix imports
	f=$(grep -rl 'github.com/hashicorp/consul/command/agent' * | grep '\.go')
	gsed -i -e 's|github.com/hashicorp/consul/command/agent|github.com/hashicorp/consul/agent|' $f
	goimports -w $f

	f=$(grep -rl 'github.com/hashicorp/consul/consul' * | grep '\.go')
	gsed -i -e 's|github.com/hashicorp/consul/consul|github.com/hashicorp/consul/agent/consul|' $f
	goimports -w $f

	goimports -w command/*.go main.go
)
2017-06-10 18:52:45 +02:00
James Phillips e9ee136baf
Makes test target fail based on return code from go test. 2017-06-02 15:09:07 -07:00
Frank Schroeder 68cac3d40c
Revert "test: run all tests together again"
This reverts commit e48df7b572.
2017-06-02 07:52:21 +02:00
Frank Schroeder e48df7b572
test: run all tests together again
Looks like I have eliminated the troublesome tests for now.
2017-05-31 10:00:10 +02:00
Frank Schroeder 205cbe5fc8
build: update makefile
* capture all output to test.log only and print only failures
* always print output on travis
* test github.com/hashicorp/consul/consul separately until we've
  found the interactions with the other packages during testing.
2017-05-31 00:29:28 +02:00
Frank Schroeder 53fd15070d
build: add target for running tests with race detector 2017-05-31 00:29:25 +02:00
Frank Schroeder 2a39c480b9
test: capture stderr in test.log 2017-05-31 00:29:18 +02:00
Frank Schroeder 9993095448 test: simplify makefile 2017-05-12 22:12:47 +02:00
Frank Schroeder 57622c518a
Add make target for linux binary 2017-05-04 13:31:56 +02:00
James Phillips f37303401a Merge pull request #2987 from sean-/b-fix-dev-paths
Fix `make dev` when the `bin/` directory doesn't exist.
2017-04-28 16:44:25 -07:00
Sean Chittenden 047576148e
Fix `make dev` when the `bin/` directory doesn't exist.
Without this patch you end up with a nifty consul build named `bin`,
not `bin/consul`.
2017-04-28 16:40:50 -07:00
Frank Schroeder 3cc7a55a9a
build: Use bash for building 2017-04-27 11:49:43 -07:00
Frank Schroeder 76b9e257e3
build: Do not swallow go test exit code 2017-04-27 11:17:58 -07:00
Frank Schroeder 9c632e4eb7 build: Simplify make test and log output 2017-04-27 10:34:30 -07:00
Frank Schröder b6ae9f187c Faster dev builds (#2924)
This patch runs 'go install' instead of gox which runs 'go build' for
'make dev' and copies the binary into ./bin and ./pkg/${GOOS}_${GOARCH}
to mimick the previous behavior. This reduces the roundtrip times
for a dev build from 11 sec to 500ms if there weren't any changes.
2017-04-18 15:03:51 -07:00
Seth Vargo 0a04d23f50
Modernize makefile a bit 2017-03-23 20:29:32 -04:00
Sean Chittenden 35fd064a79
Revert `97904cec95f1a1fafd18405f0dd143eb63d595a9` 2017-02-01 16:32:30 -08:00
Sean Chittenden 97904cec95
Catch cross-compiliation build failures at Travis. 2017-02-01 10:22:27 -08:00
James Phillips bc47511d26
Runs the static asset packaging inside the container; updates assets. 2016-11-08 15:14:08 -08:00
James Phillips 9f2c965a63
Makes vendor pattern more generic. 2016-11-02 17:33:56 -07:00
James Phillips 2a950f3897 Adds basic build tag support with different versions. (#2463) 2016-11-02 17:27:49 -07:00
James Phillips 1468930c93 Adds a UI build in a Docker container. (#2462) 2016-11-02 15:53:02 -07:00
James Phillips 5a06d5eefc
Only builds the binary stuff under master branch CI. 2016-10-25 13:49:57 -07:00
Sean Chittenden 3fc714b9b6 Add `tools/cmd/cover` to GOTOOLS 2016-05-07 13:02:12 -07:00
Sean Chittenden e59509227c golang.org/x/tools/cmd/vet is obsolete after Go1.5
`go tool vet` is builtin starting with Go 1.5
2016-05-07 13:01:07 -07:00
James Phillips 3f7bd5dec2 Tweaks some of the default makefile targets. 2016-02-17 20:36:48 -08:00
James Phillips baef89a8f6 Removes from cruft from the makefile. 2016-02-17 20:30:06 -08:00
James Phillips 6643c94b11 Makes vet check more sure fire now that it's xargs-ed. 2016-02-17 16:02:42 -08:00
Sean Chittenden 0c8b66ddca Don't run `go vet` on vendor/ 2016-02-12 18:52:40 -08:00
Sean Chittenden 1071bc271a Add a tools target that fetches various build-time tools 2016-02-12 17:09:18 -08:00
Sean Chittenden 25b09713b9 Manage dependencies via Godep
Embrace the future and use Go 1.6's vendor support via Godep.

Go 1.5 users should `export GO15VENDOREXPERIMENT=1`
2016-02-12 16:50:37 -08:00
Sean Chittenden 71d02d2a32 Acknowledge that we're using GNU make's dialect and rename appropriate
Makefiles to GNUmakefiles so that non-GNU make(1) will error out.  This
should be a transparent change to the universe of people using GNU make.
2016-02-05 14:24:26 -08:00