2 Commits

Author	SHA1	Message	Date
Michael Bradley, Jr	991d958436	build: merge test_dapps with the root monorepo ... [PR 1318][PR1318] introduces a monorepo member that's used as a DApp dependency, but the present arrangement whereby `test_dapps/` is an embedded monorepo makes it difficult to develop and test such packages in tandem with changes to `test_dapps/packages/*`. Reorganize `test_dapps/*` as members of the root monorepo and make various adjustments accordingly. This makes it easy to develop test dapps and any packages they depend on simultaneously, but we lose the CI/QA arrangement where test dapps are run with an embark installed from fresh tarballs. That arrangement, which is quite worthwhile as a means to detect problems arising from transitive dependencies as soon as possible, will be re-introduced in another PR, possibly involving an auxiliary repository such as embark-framework/dapp-bin. Since the `package.json` `"test"` scripts of `test_dapps/*` are now kicked off as part of `yarn test` in the root, and since port allocation isn't fully dynamic, it's important to run `yarn test` with `lerna run`'s `--concurrency=1` option. For the same reasons, the root `ci` and `qa` scripts are similarly restricted. In the future, this setup can be refactored and improved, along with use of `lerna run`'s `--since` option to increase testing efficiency in CI. [PR1318]: https://github.com/embark-framework/embark/pull/1318	2019-02-12 16:59:54 -05:00
Michael Bradley, Jr	7a93e4b6a7	build: implement a monorepo with Lerna ... TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9	2019-02-04 14:28:49 -06:00