From cd6e62049ccb45e9ec1d4df6ad86854eaf2a8c74 Mon Sep 17 00:00:00 2001 From: Matthias Kadenbach Date: Sat, 11 Feb 2017 11:13:27 -0800 Subject: [PATCH] Update documentation --- CONTRIBUTING.md | 2 +- FAQ.md | 60 +++++++++++++++++++++++++++++++++ MIGRATIONS.md | 5 +++ Makefile | 5 ++- README.md | 83 ++++++++++++++++++++++++++++++---------------- cli/README.md | 81 ++++++++++++++++++++++++++++++++++++++------ database/driver.go | 23 +++++++++---- database/mariadb | 1 - source/driver.go | 22 +++++++++--- 9 files changed, 227 insertions(+), 55 deletions(-) create mode 100644 FAQ.md create mode 100644 MIGRATIONS.md delete mode 120000 database/mariadb diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 46969a2..fcf82a4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,7 +10,7 @@ 7. `make restore-import-paths` to restore import paths 8. Push code and open Pull Request -Some more notes: +Some more helpful commands: * You can specify which database/ source tests to run: `make test-short SOURCE='file go-bindata' DATABASE='postgres cassandra'` diff --git a/FAQ.md b/FAQ.md new file mode 100644 index 0000000..f847c23 --- /dev/null +++ b/FAQ.md @@ -0,0 +1,60 @@ +# FAQ + +#### How is the code base structured? + ``` + / package migrate (the heart of everything) + /cli the CLI wrapper + /database database driver and sub directories have the actual driver implementations + /source source driver and sub directories have the actual driver implementations + ``` + +#### Why is there no `source/driver.go:Last()`? + It's not needed. And unless the source has a "native" way to read a directory in reversed order, + it might be expensive to do a full directory scan in order to get the last element. + +#### What is a NilMigration? NilVersion? + @TODO + +#### What is the difference between uint(version) and int(targetVersion)? + version refers to an existing migration version coming from a source and therefor can never be negative. + targetVersion can either be a version OR represent a NilVersion, which equals -1. + +#### What's the difference between Next/Previous and Up/Down? + ``` + 1_first_migration.up next -> 2_second_migration.up ... + 1_first_migration.down <- previous 2_second_migration.down ... + ``` + +#### Why two separate files (up and down) for a migration? + It makes all of our lives easier. No new markup/syntax to learn for users + and existing database utility tools continue to work as expected. + +#### How many migrations can migrate handle? + Whatever the maximum positive signed integer value is for your platform. + For 32bit it would be 2,147,483,647 migrations. Migrate only keeps references to + the currently run and pre-fetched migrations in memory. Please note that some + source drivers need to do build a full "directory" tree first, which puts some + heat on the memory consumption. + +#### Are the table tests in migrate_test.go bloated? + Yes and no. There are duplicate test cases for sure but they don't hurt here. In fact + the tests are very visual now and might help new users understand expected behaviors quickly. + Migrate from version x to y and y is the last migration? Just check out the test for + that particular case and know what's going on instantly. + +#### What is Docker being used for? + Only for testing. See [testing/docker.go](testing/docker.go) + +#### Why not just use docker-compose? + It doesn't give us enough runtime control for testing. We want to be able to bring up containers fast + and whenever we want, not just once at the beginning of all tests. + +#### Can I maintain my driver in my own repository? + Yes, technically thats possible. We want to encourage you to contribute your driver to this respository though. + The driver's functionality is dictated by migrate's interfaces. That means there should really + just be one driver for a database/ source. We want to prevent a future where several drivers doing the exact same thing, + just implemented a bit differently, co-exist somewhere on Github. If users have to do research first to find the + "best" available driver for a database in order to get started, we would have failed as an open source community. + +#### Can I mix multiple sources during a batch of migrations? + No. diff --git a/MIGRATIONS.md b/MIGRATIONS.md new file mode 100644 index 0000000..797fe44 --- /dev/null +++ b/MIGRATIONS.md @@ -0,0 +1,5 @@ +# Migrations + +## Best practices: How to write migrations. + +@TODO diff --git a/Makefile b/Makefile index b49ac6d..945b61d 100644 --- a/Makefile +++ b/Makefile @@ -106,9 +106,8 @@ release: define external_deps - @echo -- $(1) - @go list -f '{{join .Deps "\n"}}' $(1) | grep -v github.com/$(REPO_OWNER)/migrate | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}' - @#\n + @echo '-- $(1)'; go list -f '{{join .Deps "\n"}}' $(1) | grep -v github.com/$(REPO_OWNER)/migrate | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}' + endef diff --git a/README.md b/README.md index d36e084..bb09ed6 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,23 @@ -# migrate - [![Build Status](https://travis-ci.org/mattes/migrate.svg?branch=v3.0-prev)](https://travis-ci.org/mattes/migrate) [![GoDoc](https://godoc.org/github.com/mattes/migrate?status.svg)](https://godoc.org/github.com/mattes/migrate) [![Coverage Status](https://coveralls.io/repos/github/mattes/migrate/badge.svg?branch=v3.0-prev)](https://coveralls.io/github/mattes/migrate?branch=v3.0-prev) [![packagecloud.io](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/mattes/migrate?filter=debs) -__Database migrations written in Go. Use as CLI or import as library.__ +# migrate + +__Database migrations written in Go. Use as [CLI](#cli-usage) or import as [library](#use-in-your-go-project).__ + + * Migrate reads migrations from [sources](#migration-sources) + and applies them in correct order to a [database](#databases). + * Drivers are "dumb", migrate glues everything together and makes sure the logic is bulletproof. + (Keeps the drivers lightweight, too.) + * Database drivers don't assume things or try to correct user input. When in doubt, fail. + ## Databases -Database drivers are responsible for applying migrations to databases. -Implementing a new database driver is easy. Just implement [database/driver interface](database/driver.go) +Database drivers run migrations. [Add a new database?](database/driver.go) * [PostgreSQL](database/postgres) * [Cassandra](database/cassandra) @@ -24,10 +30,10 @@ Implementing a new database driver is easy. Just implement [database/driver inte * [Shell](database/shell) + ## Migration Sources -Source Drivers read migrations from various locations. Implementing a new source driver -is easy. Just implement the [source/driver interface](source/driver.go). +Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go) * [Filesystem](source/file) - read from fileystem (always included) * [Go-Bindata](source/go-bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata)) @@ -36,56 +42,75 @@ is easy. Just implement the [source/driver interface](source/driver.go). * [Google Cloud Storage](source/google-cloud-storage) - read from Google Cloud Platform Storage + ## CLI usage -__[CLI Documentation](cli/README.md)__ + * Simple wrapper around this library. + * Handles ctrl+c (SIGINT) gracefully. + * No config search paths, no config files, no magic ENV var injections. - -Example: +__[CLI Documentation](cli)__ ``` -go get -u -tags 'postgres' -o migrate github.com/mattes/migrate/cli - -migrate -database postgres://localhost:5432/database up 2 +$ brew install migrate --with-postgres +$ migrate -database postgres://localhost:5432/database up 2 ``` ## Use in your Go project + * API is stable and frozen for this release (v3.x). + * Package migrate has no external dependencies. + * Only import the drivers you need. + (check [dependency_tree.txt](https://github.com/mattes/migrate/releases) for each driver) + * To help prevent database corruptions, it supports graceful stops via `GracefulStop chan bool`. + * Bring your own logger. + * Uses `io.Reader` streams internally for low memory overhead. + * Thread-safe. + +__[Go Documentation](https://godoc.org/github.com/mattes/migrate)__ + ```go import ( - "github.com/mattes/migrate/migrate" - _ "github.com/mattes/migrate/database/postgres" - _ "github.com/mattes/migrate/source/github" + "github.com/mattes/migrate/migrate" + _ "github.com/mattes/migrate/database/postgres" + _ "github.com/mattes/migrate/source/github" ) func main() { - m, err := migrate.New("github://mattes:personal-access-token@mattes/migrate_test", - "postgres://localhost:5432/database?sslmode=enable") - m.Steps(2) + m, err := migrate.New( + "github://mattes:personal-access-token@mattes/migrate_test", + "postgres://localhost:5432/database?sslmode=enable") + m.Steps(2) } ``` ## Migration files -Each migration version has an up and down migration. +Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-files-up-and-down-for-a-migration) ``` 1481574547_create_users_table.up.sql 1481574547_create_users_table.down.sql ``` -## Development, Testing and Contributing - -__[Guide](CONTRIBUTING.md)__ +[Best practices: How to write migrations.](MIGRATIONS.md) -## Alternatives - * https://bitbucket.org/liamstask/goose - * https://github.com/tanel/dbmigrate - * https://github.com/BurntSushi/migration - * https://github.com/DavidHuie/gomigrate - * https://github.com/rubenv/sql-migrate +## Development and Contributing + +Yes, please! [`Makefile`](Makefile) is your friend, +read the [development guide](CONTRIBUTING.md). + +Also have a look at the [FAQ](FAQ.md). + +--- + +__Alternatives__ + +https://bitbucket.org/liamstask/goose, https://github.com/tanel/dbmigrate, +https://github.com/BurntSushi/migration, https://github.com/DavidHuie/gomigrate, +https://github.com/rubenv/sql-migrate diff --git a/cli/README.md b/cli/README.md index f3b6156..07819b0 100644 --- a/cli/README.md +++ b/cli/README.md @@ -1,13 +1,36 @@ -# CLI +# migrate CLI ## Installation +#### With Go toolchain + ``` -# dowload, build and install the CLI tool -# -tags takes database and source drivers and will only build those $ go get -u -tags 'postgres' -o migrate github.com/mattes/migrate/cli ``` +#### MacOS + +``` +$ brew install migrate --with-postgres +``` + +#### Linux (with deb package) + +``` +# TODO: add key and repo +$ apt-get update +$ apt-get install migrate +``` + +#### Download pre-build binary (Windows, MacOS, or Linux) + +[Release Downloads](https://github.com/mattes/migrate/releases) + +``` +$ curl -L https://github.com/mattes/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz +``` + + ## Usage @@ -31,13 +54,49 @@ Commands: down [N] Apply all or N down migrations drop Drop everyting inside database version Print current migration version - - -# so let's say you want to run the first two migrations -migrate -database postgres://localhost:5432/database up 2 - -# if your migrations are hosted on github -migrate -source github://mattes:personal-access-token@mattes/migrate_test \ - -database postgres://localhost:5432/database down 2 ``` + +So let's say you want to run the first two migrations + +``` +$ migrate -database postgres://localhost:5432/database up 2 +``` + +If your migrations are hosted on github + +``` +$ migrate -source github://mattes:personal-access-token@mattes/migrate_test \ + -database postgres://localhost:5432/database down 2 +``` + +The CLI will gracefully stop at a safe point when SIGINT (ctrl+c) is received. +Send SIGKILL for immediate halt. + + + +## Reading CLI arguments from somewhere else + +##### ENV variables + +``` +$ migrate -database "$MY_MIGRATE_DATABASE" +``` + +##### JSON files + +Check out https://stedolan.github.io/jq/ + +``` +$ migrate -database "$(cat config.json | jq '.database')" +``` + +##### YAML files + +```` +$ migrate -database "$(cat config/database.yml | ruby -ryaml -e "print YAML.load(STDIN.read)['database']")" +$ migrate -database "$(cat config/database.yml | python -c 'import yaml,sys;print yaml.safe_load(sys.stdin)["database"]')" +``` + + + diff --git a/database/driver.go b/database/driver.go index 53c9d8f..30b8051 100644 --- a/database/driver.go +++ b/database/driver.go @@ -20,13 +20,24 @@ const NilVersion int = -1 var driversMu sync.RWMutex var drivers = make(map[string]Driver) -// Driver is an interface every driver must implement. -// The driver implementation must pass the `Test` in database/testing. -// Optionally provide a `WithInstance` function, so users can bypass `Open` -// and use an existing database instance. +// Driver is the interface every database driver must implement. // -// Implementations must not assume things nor try to correct user input. -// If in doubt, return an error. +// How to implement a database driver? +// 1. Implement this interface. +// 2. Optionally, add a function named `WithInstance`. +// This function should accept an existing DB instance and a Config{} struct +// and return a driver instance. +// 3. Add a test that calls database/testing.go:Test() +// 4. Add own tests for Open(), WithInstance() (when provided) and Close(). +// All other functions are tested by tests in database/testing. +// Saves you some time and makes sure all database drivers behave the same way. +// 5. Call Register in init(). +// +// Guidelines: +// * Don't try to correct user input. Don't assume things. +// When in doubt, return an error and explain the situation to the user. +// * All configuration input must come from the URL string in func Open() +// or the Config{} struct in WithInstance. Don't os.Getenv(). type Driver interface { // Open returns a new driver instance configured with parameters // coming from the URL string. Migrate will call this function diff --git a/database/mariadb b/database/mariadb deleted file mode 120000 index 0d46ca3..0000000 --- a/database/mariadb +++ /dev/null @@ -1 +0,0 @@ -mysql \ No newline at end of file diff --git a/source/driver.go b/source/driver.go index 3d7abb1..103138d 100644 --- a/source/driver.go +++ b/source/driver.go @@ -14,10 +14,24 @@ import ( var driversMu sync.RWMutex var drivers = make(map[string]Driver) -// Driver is an interface every driver must implement. -// The driver implementation must pass the `Test` in source/testing. -// Optionally provide a `WithInstance` function, so users can bypass `Open` -// and use an existing source instance. +// Driver is the interface every source driver must implement. +// +// How to implement a source driver? +// 1. Implement this interface. +// 2. Optionally, add a function named `WithInstance`. +// This function should accept an existing source instance and a Config{} struct +// and return a driver instance. +// 3. Add a test that calls source/testing.go:Test() +// 4. Add own tests for Open(), WithInstance() (when provided) and Close(). +// All other functions are tested by tests in source/testing. +// Saves you some time and makes sure all source drivers behave the same way. +// 5. Call Register in init(). +// +// Guidelines: +// * All configuration input must come from the URL string in func Open() +// or the Config{} struct in WithInstance. Don't os.Getenv(). +// * Drivers are supposed to be read only. +// * Ideally don't load any contents (into memory) in Open or WithInstance. type Driver interface { // Open returns a a new driver instance configured with parameters // coming from the URL string. Migrate will call this function