embark

Commit Graph

Author	SHA1	Message	Date
Michael Bradley, Jr	3693ebd90d	fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).	2020-02-25 14:52:10 -06:00
Michael Bradley, Jr	7a93e4b6a7	build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For test releases (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's not implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9	2019-02-04 14:28:49 -06:00
Michael Bradley, Jr	f10e0258cb	build: introduce a `prepare` script in embark's package.json TL;DR These changes affect workflow with yarn. To prevent embark's `prepare` script from running undesirably: - If node_modules is in place and you're reinstalling after switching branches: ``` yarn run install_all ``` - If node_modules is missing (fresh clone or deleted): ``` EMBARK_NO_PREPARE=t yarn install && yarn run install_all ``` It's not recommended to set `EMBARK_NO_PREPARE` in your environment (e.g. in `.bashrc`) since that would interfere with embark's `release` script if/when you run it. ----------------- 1. Specify embark's build-related steps in the `prepare` script of package.json. When embark is installed directly from GitHub the `prepare` script results in a "pre install" phase (handled automatically by npm/yarn) that fetches devDependencies, builds embark (including embark-ui), packs a tarball with the same steps (minus testing and tree-checking) as would happen during an embark release, and finally does a production install from that tarball. Important point: installs from GitHub must be performed with yarn; they're no longer possible with npm since during the "pre install" phase npm will honor embark's `.npmrc` and `"engines"` settings. The following will work correctly after this commit is merged: ``` yarn [global] add git+https://github.com/embark-framework/embark.git ``` Use of "hosted git" shortcuts (e.g. `embark-framework/embark#bracnh`) won't work correctly because yarn doesn't fully support them. See: https://github.com/yarnpkg/yarn/issues/5235. It's important to use `git+https` urls. Following a succesful install with `git+https` it is possible to use a "hosted git" shortcut or `https` url, but that's owing to a subtle and unreliable interaction between yarn's cache and yarn's logic for installing from a url/shortcut. 2. Adjust the npm configs (`.npmrc`) for embark/-ui so that `yarn run [cmd] [--opt]` can be used in place of `npm run [cmd] -- [--opt]`. Either way is okay for running scripts, they're equivalent, but note the requirement to use `--` before specifying command options with `npm run`. 3. Introduce yarn configs (`.yarnrc`) for embark/-ui and include the `check-files` directive. H/t to @alaibe for the recommendation. 4. Ignore embark's `dist/typings` and `scripts` directories when packing a tarball. 5. Refactor embark/-ui's npm-scripts in relation to the `prepare` script, and make other small improvements. Notably, if the environment variable `EMBARK_NO_PREPARE` is truthy (from JS perspective) then embark's `prepare` script will exit early. This prevents `install_all` and `prepare` from getting stuck in a loop (`install:core` uses cross-env to set `EMBARK_NO_PREPARE`) and provides a mechanism for users to skip the `prepare` script when doing a fresh install: ``` EMBARK_NO_PREPARE=t yarn install ``` 6. Give `.js` extensions to node scripts in embark's `scripts/`, remove the shebang lines, and have npm-scripts explicitly invoke them with node. This arrangement works for all platforms: Linux, macOS, and Windows. 7. Adjust travis and appveyor configs. Since at present there aren't any tests or other CI steps that make use of embark-ui's production build, set `EMBARK_NO_PREPARE` in the CI environments and invoke `build:node` directly. Check the working tree after `yarn install` for embark/-ui. This detects situations where changes should have been committed to `yarn.lock` but were not. Check the working tree again at the end to detect situations where ignore files should have been adjusted but were not. Both checks could also detect other surprising behavior that needs to be investigated. Any time the working tree is not clean (there are untracked files or changes) CI will fail. Drop CI runs for node 8.11.3 because that version ships with an older npm that results in unstaged changes to the test apps' `package-lock.json` files, causing the working tree check to fail at the end of the CI run. A simple workaround isn't apparent, but the matter can be revisited. 8. Refactor embark's `release` script in light of the `prepare` script. Notably, do the push step only after `npm publish` completes successfully. This allows embark's `prepare` and `prepublishOnly` scripts to detect problems before a commit and tag are pushed to GitHub, avoiding a need to rebase/revert the remote release branch; the local branch will still need to have a commit dropped and tag deleted before rerunning the `release` script. Prompt the user if the `release` script is not being run in `--dry-run` mode. Provide additional visual indicators of `--dry-run` mode. Force the user to supply `--repo-branch [branch]` if the intention is to release from a branch other than `master`.	2018-12-03 16:24:10 -06:00

Author

SHA1

Message

Date

Michael Bradley, Jr

3693ebd90d

fix: ensure that packages properly specify their dependencies

Many packages in the monorepo did not specify all of their dependencies; they
were effectively relying on resolution in the monorepo's root
`node_modules`. In a production release of `embark` and `embark[js]-*` packages
this can lead to broken packages.

To fix the problem currently and to help prevent it from happening again, make
use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies`
and `import/no-unresolved` rules. In the root `tslint.json` set
`"no-implicit-dependencies": true`, wich is the tslint equivalent of
`import/no-extraneous-dependencies`; there is no tslint equivalent for
`import/no-unresolved`, but we will eventually replace tslint with an eslint
configuration that checks both `.js` and `.ts` files.

For `import/no-unresolved` to work in our monorepo setup, in most packages add
an `index.js` that has:

```js
module.exports = require('./dist'); // or './dist/lib' in some cases
```

And point `"main"` in `package.json` to `"./index.js"`. Despite what's
indicated in npm's documentation for `package.json`, it's also necessary to add
`"index.js"` to the `"files"` array.

Make sure that all `.js` files that can and should be linted are in fact
linted. For example, files in `packages/embark/src/cmd/` weren't being linted
and many test suites weren't being linted.

Bump all relevant packages to `eslint@6.8.0`.

Fix all linter errors that arose after these changes.

Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and
`"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the
monorepo. The script exits with error if any specifiers are found in
`yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions,
cf. `scripts/check-yarn-lock.js`).

2020-02-25 14:52:10 -06:00

Michael Bradley, Jr

7a93e4b6a7

build: implement a monorepo with Lerna

TL;DR
=====

`yarn install` in a fresh clone of the repo.

`yarn reboot` when switching branches.

When pulling in these changes, there may be untracked files at the root in
all/some of:

```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```

They can be safely deleted since those paths are no longer in use at the root.

Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:

`yarn build --scope embark` build only `packages/embark`.

`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.

Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.

It's also possible to `cd` into a particular package and run its scripts
directly:

```
cd packages/embark && yarn watch
```

Hot Topics & Questions
======================

What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?

Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?

Scripts
=======

If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.

`yarn build`
------------

Runs babel, webpack, etc. according to a package's `build` script.

`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.

`yarn ci`
---------

Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.

Also runs the `ci` script of the embedded `test_dapps` monorepo.

`yarn clean`
------------

Runs rimraf, etc. according to a package's `clean` script.

`yarn globalize`
----------------

Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).

`yarn lint`
-----------

Runs eslint, etc. according to a package's `lint` script.

`yarn package`
--------------

Invokes `npm pack` according to a package's `package` script.

`yarn qa`
---------

Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.

There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.

The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.

`yarn reboot`
-------------

Invokes the `reset` script and then does `yarn install`.

The `reboot:full` variant invokes `reset:full` and then does `yarn install`.

`yarn release`
--------------

Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).

```
yarn release [bump] [--options]
```

* `[bump]` see [`publish` positionals][pub-pos] and [`version`
  positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
  will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
  by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
  [`lerna.json`][lerna-json].

To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:

```
yarn release prerelease --dist-tag next
```

For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.

Condensend instructions:

```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```

`yarn reset`
------------

Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.

The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.

Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.

`yarn start`
------------

Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.

`yarn test`
-----------

Run mocha, etc. according to a package's `test` script.

The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.

`yarn test_dapps`
-----------------

Runs the `test` script of the embedded `test_dapps` monorepo.

The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.

`yarn typecheck`
----------------

Runs tsc, etc. according to a package's `typecheck` script.

Notes
=====

`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.

[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.

All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.

`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).

Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.

In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.

The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.

Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.

`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.

`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".

`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.

`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.

[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9

2019-02-04 14:28:49 -06:00

Michael Bradley, Jr

f10e0258cb

build: introduce a `prepare` script in embark's package.json

**TL;DR**

These changes affect workflow with yarn. To prevent embark's `prepare` script
from running undesirably:

- If node_modules is in place and you're reinstalling after switching branches:

```
yarn run install_all
```

- If node_modules is missing (fresh clone or deleted):

```
EMBARK_NO_PREPARE=t yarn install && yarn run install_all
```

It's not recommended to set `EMBARK_NO_PREPARE` in your environment (e.g. in
`.bashrc`) since that would interfere with embark's `release` script if/when
you run it.

-----------------

**1.** Specify embark's build-related steps in the `prepare` script of
  package.json.

When embark is installed directly from GitHub the `prepare` script results in a
"pre install" phase (handled automatically by npm/yarn) that fetches
devDependencies, builds embark (including embark-ui), packs a tarball with the
same steps (minus testing and tree-checking) as would happen during an embark
release, and finally does a production install from that tarball.

Important point: installs from GitHub must be performed with yarn; they're no
longer possible with npm since during the "pre install" phase npm will honor
embark's `.npmrc` and `"engines"` settings.

The following will work correctly after this commit is merged:

```
yarn [global] add git+https://github.com/embark-framework/embark.git
```

Use of "hosted git" shortcuts (e.g. `embark-framework/embark#bracnh`) won't
work correctly because yarn doesn't fully support them. See:
https://github.com/yarnpkg/yarn/issues/5235.

It's important to use `git+https` urls. Following a succesful install with
`git+https` it is possible to use a "hosted git" shortcut or `https` url, but
that's owing to a subtle and unreliable interaction between yarn's cache and
yarn's logic for installing from a url/shortcut.

**2.** Adjust the npm configs (`.npmrc`) for embark/-ui so that `yarn run [cmd]
  [--opt]` can be used in place of `npm run [cmd] -- [--opt]`.

Either way is okay for running scripts, they're equivalent, but note the
requirement to use `--` before specifying command options with `npm run`.

**3.** Introduce yarn configs (`.yarnrc`) for embark/-ui and include the
  `check-files` directive.

H/t to @alaibe for the recommendation.

**4.** Ignore embark's `dist/typings` and `scripts` directories when packing a
  tarball.

**5.** Refactor embark/-ui's npm-scripts in relation to the `prepare` script,
  and make other small improvements.

Notably, if the environment variable `EMBARK_NO_PREPARE` is truthy (from JS
perspective) then embark's `prepare` script will exit early. This prevents
`install_all` and `prepare` from getting stuck in a loop (`install:core` uses
cross-env to set `EMBARK_NO_PREPARE`) and provides a mechanism for users to
skip the `prepare` script when doing a fresh install:

```
EMBARK_NO_PREPARE=t yarn install
```

**6.** Give `.js` extensions to node scripts in embark's `scripts/`, remove the
  shebang lines, and have npm-scripts explicitly invoke them with node.

This arrangement works for all platforms: Linux, macOS, and Windows.

**7.** Adjust travis and appveyor configs.

Since at present there aren't any tests or other CI steps that make use of
embark-ui's production build, set `EMBARK_NO_PREPARE` in the CI environments
and invoke `build:node` directly.

Check the working tree after `yarn install` for embark/-ui. This detects
situations where changes should have been committed to `yarn.lock` but were
not. Check the working tree again at the end to detect situations where ignore
files should have been adjusted but were not. Both checks could also detect
other surprising behavior that needs to be investigated. Any time the working
tree is not clean (there are untracked files or changes) CI will fail.

Drop CI runs for node 8.11.3 because that version ships with an older npm that
results in unstaged changes to the test apps' `package-lock.json` files,
causing the working tree check to fail at the end of the CI run. A simple
workaround isn't apparent, but the matter can be revisited.

**8.** Refactor embark's `release` script in light of the `prepare` script.

Notably, do the push step only after `npm publish` completes successfully. This
allows embark's `prepare` and `prepublishOnly` scripts to detect problems
before a commit and tag are pushed to GitHub, avoiding a need to rebase/revert
the remote release branch; the local branch will still need to have a commit
dropped and tag deleted before rerunning the `release` script.

Prompt the user if the `release` script is not being run in `--dry-run` mode.

Provide additional visual indicators of `--dry-run` mode.

Force the user to supply `--repo-branch [branch]` if the intention is to
release from a branch other than `master`.

2018-12-03 16:24:10 -06:00

3 Commits