Michael Bradley, Jr 7a93e4b6a7 build: implement a monorepo with Lerna
TL;DR
=====

`yarn install` in a fresh clone of the repo.

`yarn reboot` when switching branches.

When pulling in these changes, there may be untracked files at the root in
all/some of:

```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```

They can be safely deleted since those paths are no longer in use at the root.

Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:

`yarn build --scope embark` build only `packages/embark`.

`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.

Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.

It's also possible to `cd` into a particular package and run its scripts
directly:

```
cd packages/embark && yarn watch
```

Hot Topics & Questions
======================

What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?

Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?

Scripts
=======

If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.

`yarn build`
------------

Runs babel, webpack, etc. according to a package's `build` script.

`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.

`yarn ci`
---------

Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.

Also runs the `ci` script of the embedded `test_dapps` monorepo.

`yarn clean`
------------

Runs rimraf, etc. according to a package's `clean` script.

`yarn globalize`
----------------

Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).

`yarn lint`
-----------

Runs eslint, etc. according to a package's `lint` script.

`yarn package`
--------------

Invokes `npm pack` according to a package's `package` script.

`yarn qa`
---------

Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.

There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.

The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.

`yarn reboot`
-------------

Invokes the `reset` script and then does `yarn install`.

The `reboot:full` variant invokes `reset:full` and then does `yarn install`.

`yarn release`
--------------

Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).

```
yarn release [bump] [--options]
```

* `[bump]` see [`publish` positionals][pub-pos] and [`version`
  positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
  will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
  by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
  [`lerna.json`][lerna-json].

To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:

```
yarn release prerelease --dist-tag next
```

For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.

Condensend instructions:

```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```

`yarn reset`
------------

Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.

The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.

Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.

`yarn start`
------------

Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.

`yarn test`
-----------

Run mocha, etc. according to a package's `test` script.

The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.

`yarn test_dapps`
-----------------

Runs the `test` script of the embedded `test_dapps` monorepo.

The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.

`yarn typecheck`
----------------

Runs tsc, etc. according to a package's `typecheck` script.

Notes
=====

`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.

[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.

All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.

`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).

Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.

In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.

The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.

Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.

`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.

`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".

`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.

`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.

[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-02-04 14:28:49 -06:00

188 lines
5.8 KiB
JavaScript

// Do this as the first thing so that any code reading it knows the right env.
process.env.BABEL_ENV = 'production';
process.env.NODE_ENV = 'production';
// Makes the script crash on unhandled rejections instead of silently
// ignoring them. In the future, promise rejections that are not handled will
// terminate the Node.js process with a non-zero exit code.
process.on('unhandledRejection', err => {
throw err;
});
// Ensure environment variables are read.
require('../config/env');
const path = require('path');
const chalk = require('chalk');
const fs = require('fs-extra');
const webpack = require('webpack');
const bfj = require('bfj');
const config = require('../config/webpack.config.prod');
const paths = require('../config/paths');
const checkRequiredFiles = require('react-dev-utils/checkRequiredFiles');
const formatWebpackMessages = require('react-dev-utils/formatWebpackMessages');
const printHostingInstructions = require('react-dev-utils/printHostingInstructions');
const FileSizeReporter = require('react-dev-utils/FileSizeReporter');
const printBuildError = require('react-dev-utils/printBuildError');
const measureFileSizesBeforeBuild =
FileSizeReporter.measureFileSizesBeforeBuild;
const printFileSizesAfterBuild = FileSizeReporter.printFileSizesAfterBuild;
const useYarn = fs.existsSync(paths.yarnLockFile);
// These sizes are pretty large. We'll warn for bundles exceeding them.
const WARN_AFTER_BUNDLE_GZIP_SIZE = 512 * 1024;
const WARN_AFTER_CHUNK_GZIP_SIZE = 1024 * 1024;
const isInteractive = process.stdout.isTTY;
// Warn and crash if required files are missing
if (!checkRequiredFiles([paths.appHtml, paths.appIndexJs])) {
process.exit(1);
}
// Process CLI arguments
const argv = process.argv.slice(2);
const writeStatsJson = argv.indexOf('--stats') !== -1;
// We require that you explicitly set browsers and do not fall back to
// browserslist defaults.
const { checkBrowsers } = require('react-dev-utils/browsersHelper');
checkBrowsers(paths.appPath, isInteractive)
.then(() => {
// First, read the current file sizes in build directory.
// This lets us display how much they changed later.
return measureFileSizesBeforeBuild(paths.appBuild);
})
.then(previousFileSizes => {
// Remove all content but keep the directory so that
// if you're in it, you don't end up in Trash
fs.emptyDirSync(paths.appBuild);
// Merge with the public folder
copyPublicFolder();
// Start the webpack build
return build(previousFileSizes);
})
.then(
({ stats, previousFileSizes, warnings }) => {
if (warnings.length) {
console.log(chalk.yellow('Compiled with warnings.\n'));
console.log(warnings.join('\n\n'));
console.log(
'\nSearch for the ' +
chalk.underline(chalk.yellow('keywords')) +
' to learn more about each warning.'
);
console.log(
'To ignore, add ' +
chalk.cyan('// eslint-disable-next-line') +
' to the line before.\n'
);
} else {
console.log(chalk.green('Compiled successfully.\n'));
}
console.log('File sizes after gzip:\n');
printFileSizesAfterBuild(
stats,
previousFileSizes,
paths.appBuild,
WARN_AFTER_BUNDLE_GZIP_SIZE,
WARN_AFTER_CHUNK_GZIP_SIZE
);
console.log();
const appPackage = require(paths.appPackageJson);
const publicUrl = paths.publicUrl;
const publicPath = config.output.publicPath;
const buildFolder = path.relative(process.cwd(), paths.appBuild);
printHostingInstructions(
appPackage,
publicUrl,
publicPath,
buildFolder,
useYarn
);
},
err => {
console.log(chalk.red('Failed to compile.\n'));
printBuildError(err);
process.exit(1);
}
)
.catch(err => {
if (err && err.message) {
console.log(err.message);
}
process.exit(1);
});
// Create the production build and print the deployment instructions.
function build(previousFileSizes) {
console.log('Creating an optimized production build...');
let compiler = webpack(config);
return new Promise((resolve, reject) => {
compiler.run((err, stats) => {
let messages;
if (err) {
if (!err.message) {
return reject(err);
}
messages = formatWebpackMessages({
errors: [err.message],
warnings: [],
});
} else {
messages = formatWebpackMessages(
stats.toJson({ all: false, warnings: true, errors: true })
);
}
if (messages.errors.length) {
// Only keep the first error. Others are often indicative
// of the same problem, but confuse the reader with noise.
if (messages.errors.length > 1) {
messages.errors.length = 1;
}
return reject(new Error(messages.errors.join('\n\n')));
}
if (
process.env.CI &&
(typeof process.env.CI !== 'string' ||
process.env.CI.toLowerCase() !== 'false') &&
messages.warnings.length
) {
console.log(
chalk.yellow(
'\nTreating warnings as errors because process.env.CI = true.\n' +
'Most CI servers set it automatically.\n'
)
);
return reject(new Error(messages.warnings.join('\n\n')));
}
const resolveArgs = {
stats,
previousFileSizes,
warnings: messages.warnings,
};
if (writeStatsJson) {
return bfj
.write(paths.appBuild + '/bundle-stats.json', stats.toJson())
.then(() => resolve(resolveArgs))
.catch(error => reject(new Error(error)));
}
return resolve(resolveArgs);
});
});
}
function copyPublicFolder() {
fs.copySync(paths.appPublic, paths.appBuild, {
dereference: true,
filter: file => file !== paths.appHtml,
});
}