embarklabs
/
embark
mirror of https://github.com/embarklabs/embark.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
embark/package.json

88 lines
3.3 KiB
JSON
Raw Normal View History

add npm package
2015-05-24 12:38:56 +00:00
{
restructure; move code from tasks into libs
2015-06-28 02:20:07 +00:00
"devDependencies": {
build: implement a common babel config Setup a `babel.config.js` in the root of the monorepo to be used by `packages/*`. It won't be used by some packages, e.g. `packages/embark-ui`, but most of them should use it instead of rolling their own. Allow for package-level modifications by specifying `babelrcRoots` in the root `babel.config.js` Use the babel `--root-mode upward` option in `packages/embark`'s `build` script. Other packages intending to use the common config should do likewise. Use a `.babelrc.js` in `packages/embark` to supply the package-specific `ignore` settings. Make packages used by the common config devDeps of the root. Extract babel-related devDeps from `packages/embark`, but don't extract the non-dev deps since those are used by embark's pipeline in a production install. Normally, it should only be necessary to have `@babel/cli` and `@babel/core` in devDeps, and possibly `@babel/runtime-corejs2` in deps, plus any package-specific babel-related dev/deps. Once we deprecate the pipeline, we can finish the extraction. Use `ncu -f '/babel/' -u` to bump the versions of all babel-related deps in the root and in `packages/embark`. We get better space/time savings from the yarn workspace when versions match.
2019-02-05 20:29:51 +00:00
"@babel/core": "7.2.2",
"@babel/plugin-proposal-class-properties": "7.3.0",
"@babel/plugin-proposal-decorators": "7.3.0",
"@babel/plugin-proposal-optional-chaining": "7.2.0",
"@babel/plugin-syntax-dynamic-import": "7.2.0",
"@babel/plugin-transform-runtime": "7.2.0",
build(deps): bump @babel/preset-env from 7.3.1 to 7.5.5 Bumps [@babel/preset-env](https://github.com/babel/babel) from 7.3.1 to 7.5.5. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/master/CHANGELOG.md) - [Commits](https://github.com/babel/babel/compare/v7.3.1...v7.5.5) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-13 15:53:50 +00:00
"@babel/preset-env": "7.5.5",
build: implement a common babel config Setup a `babel.config.js` in the root of the monorepo to be used by `packages/*`. It won't be used by some packages, e.g. `packages/embark-ui`, but most of them should use it instead of rolling their own. Allow for package-level modifications by specifying `babelrcRoots` in the root `babel.config.js` Use the babel `--root-mode upward` option in `packages/embark`'s `build` script. Other packages intending to use the common config should do likewise. Use a `.babelrc.js` in `packages/embark` to supply the package-specific `ignore` settings. Make packages used by the common config devDeps of the root. Extract babel-related devDeps from `packages/embark`, but don't extract the non-dev deps since those are used by embark's pipeline in a production install. Normally, it should only be necessary to have `@babel/cli` and `@babel/core` in devDeps, and possibly `@babel/runtime-corejs2` in deps, plus any package-specific babel-related dev/deps. Once we deprecate the pipeline, we can finish the extraction. Use `ncu -f '/babel/' -u` to bump the versions of all babel-related deps in the root and in `packages/embark`. We get better space/time savings from the yarn workspace when versions match.
2019-02-05 20:29:51 +00:00
"@babel/preset-typescript": "7.1.0",
"babel-plugin-dynamic-import-node": "2.2.0",
"babel-plugin-macros": "2.4.5",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"chalk": "2.4.2",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"coveralls": "3.0.2",
build(deps-dev): bump eslint-plugin-react from 7.12.4 to 7.14.3 Bumps [eslint-plugin-react](https://github.com/yannickcr/eslint-plugin-react) from 7.12.4 to 7.14.3. - [Release notes](https://github.com/yannickcr/eslint-plugin-react/releases) - [Changelog](https://github.com/yannickcr/eslint-plugin-react/blob/master/CHANGELOG.md) - [Commits](https://github.com/yannickcr/eslint-plugin-react/compare/v7.12.4...v7.14.3) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-19 14:16:51 +00:00
"eslint-plugin-react": "7.14.3",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"fs-extra": "7.0.1",
build: bump lerna package version 3.13.1 -> 3.16.4
2019-08-12 16:32:18 +00:00
"lerna": "3.16.4",
build: implement a common babel config Setup a `babel.config.js` in the root of the monorepo to be used by `packages/*`. It won't be used by some packages, e.g. `packages/embark-ui`, but most of them should use it instead of rolling their own. Allow for package-level modifications by specifying `babelrcRoots` in the root `babel.config.js` Use the babel `--root-mode upward` option in `packages/embark`'s `build` script. Other packages intending to use the common config should do likewise. Use a `.babelrc.js` in `packages/embark` to supply the package-specific `ignore` settings. Make packages used by the common config devDeps of the root. Extract babel-related devDeps from `packages/embark`, but don't extract the non-dev deps since those are used by embark's pipeline in a production install. Normally, it should only be necessary to have `@babel/cli` and `@babel/core` in devDeps, and possibly `@babel/runtime-corejs2` in deps, plus any package-specific babel-related dev/deps. Once we deprecate the pipeline, we can finish the extraction. Use `ncu -f '/babel/' -u` to bump the versions of all babel-related deps in the root and in `packages/embark`. We get better space/time savings from the yarn workspace when versions match.
2019-02-05 20:29:51 +00:00
"lodash.clonedeep": "4.5.0",
chore: introduce release script This commit automates our release process. It takes care of a couple of things: - Bumps version number in package.json as needed, see below - Generates changelog based on commit history - Create release commit - Tags release commit - Pushes release commit including tags to upstream repository - Publishes release on npm There are a couple of options supported. The base command is run like this: ``` npm run release ``` This will do a minor bump and try to push to `origin` on `master`. However, this can be altered with the following options. ``` npm run release -- --dry-run ``` Can be used to perform dry run and not actually committing, tagging, pushing, publishing anything. ``` npm run release -- --release-as <something> ``` Something can be `minor`, `major`, `patch` or anything you want `foo`, `1.0.0` etc. ``` npm run release -- --prerelease alpha ``` Will create a prerelease version a la `4.0.0-alpha.x`. ``` npm run release -- --prerelease alpha --npm-dist-tag next ``` Publishes a dist tag on npm using dist tag `next` ``` npm run release -- --sign ``` Signs the release commit (you need to have PGP setup for that). ``` npm run release -- --repo-origin pascal --repo-branch foo/bar ``` Pushes the release commit into `pascal/foo/bar`.
2018-11-09 13:26:08 +00:00
"minimist": "1.2.0",
build: bump npm-run-all to the latest version
2018-11-26 18:21:35 +00:00
"npm-run-all": "4.1.5",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"nyc": "13.1.0",
build(deps): bump rimraf from 2.6.3 to 3.0.0 Bumps [rimraf](https://github.com/isaacs/rimraf) from 2.6.3 to 3.0.0. - [Release notes](https://github.com/isaacs/rimraf/releases) - [Changelog](https://github.com/isaacs/rimraf/blob/master/CHANGELOG.md) - [Commits](https://github.com/isaacs/rimraf/compare/v2.6.3...v3.0.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-20 08:23:05 +00:00
"rimraf": "3.0.0",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"semver": "5.6.0"
spec node, npm in "engines" of package.json
2018-09-18 21:28:50 +00:00
},
"engines": {
build: change the engines range for NodeJS to indicate embark is not compatible with v12.x
2019-05-24 22:17:16 +00:00
"node": ">=8.12.0 <12.0.0",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"npm": ">=6.4.1",
build: use yarn for reproducible builds and CI yarn.lock files are generated for embark and embark-ui, and their package.json files and other npm related files are updated to support and require using yarn for development of embark itself and for embark's CI.
2018-11-15 23:36:13 +00:00
"yarn": ">=1.12.3"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
},
"name": "root",
"private": true,
"scripts": {
"build": "lerna run --stream build",
"build:no-ui": "npm run build -- --ignore embark-ui",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"ci": "lerna run --ignore embark-dapp-* --stream ci",
"ci:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream ci",
"ci:full": "npm-run-all cwtree \"ci -- --concurrency={1}\" ci:dapps cwtree -- 1",
"clean": "npx lerna run --stream clean",
"clean:full": "npx npm-run-all clean clean:top",
"clean:top": "npx rimraf node_modules",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"coverage": "npm-run-all coverage:collect coverage:report",
"coverage:collect": "node scripts/coverage-collect",
"coverage:coveralls": "nyc report --reporter=text-lcov | coveralls || exit 0",
"coverage:report": "node scripts/coverage-report",
"coveralls": "npm-run-all coverage:collect coverage:coveralls",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"cwtree": "node scripts/check-working-tree",
chore: move embark site into mono repo
2019-04-18 10:33:52 +00:00
"deploy:site": "node site/deploy-site",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"globalize": "node scripts/globalize",
"lint": "lerna run --parallel lint",
"package": "lerna run --parallel package",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"postclean": "npx lerna clean --yes",
"postreboot": "yarn install",
"postreboot:full": "yarn install",
"preci:full": "yarn install",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"preclean": "npm run reset:top",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"preqa:full": "yarn install",
"qa": "lerna run --ignore embark-dapp-* --stream qa",
"qa:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream qa",
"qa:full": "npm-run-all cwtree reboot:full cwtree \"qa -- --concurrency={1}\" qa:dapps cwtree -- 1",
"reboot": "npm run clean",
"reboot:full": "npm run clean:full",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"release": "node scripts/release",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"reset": "npx lerna run --stream reset",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"reset:full": "npx npm-run-all reset reset:top",
"reset:top": "npx rimraf .nyc_output coverage",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"start": "lerna run --parallel start",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"test": "lerna run --ignore embark-dapp-* --stream test",
"test:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream test",
"test:full": "npm-run-all test test:dapps",
fix: add watch script on top level (#1320)
2019-02-08 18:22:59 +00:00
"typecheck": "lerna run --parallel typecheck",
"watch": "lerna run --parallel watch"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
},
"workspaces": {
"packages": [
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"dapps/templates/*",
"dapps/tests/*",
"packages/*"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
],
"nohoist": [
"embark/embark-test-contract-0",
build: merge test_dapps with the root monorepo [PR 1318][PR1318] introduces a monorepo member that's used as a DApp dependency, but the present arrangement whereby `test_dapps/` is an embedded monorepo makes it difficult to develop and test such packages in tandem with changes to `test_dapps/packages/*`. Reorganize `test_dapps/*` as members of the root monorepo and make various adjustments accordingly. This makes it easy to develop test dapps and any packages they depend on simultaneously, but we lose the CI/QA arrangement where test dapps are run with an embark installed from fresh tarballs. That arrangement, which is quite worthwhile as a means to detect problems arising from transitive dependencies as soon as possible, will be re-introduced in another PR, possibly involving an auxiliary repository such as embark-framework/dapp-bin. Since the `package.json` `"test"` scripts of `test_dapps/*` are now kicked off as part of `yarn test` in the root, and since port allocation isn't fully dynamic, it's important to run `yarn test` with `lerna run`'s `--concurrency=1` option. For the same reasons, the root `ci` and `qa` scripts are similarly restricted. In the future, this setup can be refactored and improved, along with use of `lerna run`'s `--since` option to increase testing efficiency in CI. [PR1318]: https://github.com/embark-framework/embark/pull/1318
2019-02-12 07:21:04 +00:00
"embark/embark-test-contract-1",
build: add nohoist entry for bootstrap package relative to dapps/templates/demo The demo's `embark.json` relies on the bootstrap package being within the demo's own `node_modules`, but in the monorepo yarn was hoisting it up to the root `node_modules`.
2019-05-15 14:27:22 +00:00
"embark-dapp-template-demo/bootstrap",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"embark-dapp-test-app/embark-dapp-test-service",
refactor(@embarkjs/web3): make web3 connector an internal plugin like the other embarkjs-* packages Effectively deprecate the `embarkjs-connector-web3` package but don't introduce a breaking change by simply not loading the plugin if it's specified in a DApp's `embark.json`. If the deprecated plugin is specified, display a message indicating the plugin was ignored and suggesting it be removed from the project's `embark.json` and `package.json`.
2019-06-12 20:26:59 +00:00
"embark-dapp-test-app/zeppelin-solidity"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
]
restructure; move code from tasks into libs
2015-06-28 02:20:07 +00:00
}
add npm package
2015-05-24 12:38:56 +00:00
}