embarklabs
/
embark
mirror of https://github.com/embarklabs/embark.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
embark/package.json

120 lines
4.8 KiB
JSON
Raw Normal View History

add npm package
2015-05-24 12:38:56 +00:00
{
restructure; move code from tasks into libs
2015-06-28 02:20:07 +00:00
"devDependencies": {
build(deps-dev): update other @storybook packages to 5.3.9 This follows on PR #2227. Satisfy new peer dependencies following the bumps: `@storybook/core`, `regenerator-runtime`. Follow storybook's instructions to add `@storybook/preset-create-react-app` as a dependency and also add the needed `.storybook/main.js` in `packages/cockpit/ui/`. Storybook is sensitive to presets being in the immediate project's `node_modules` so add a `"nohoist"` in the root `package.json` for `"embark-ui/@storybook/**"`. After making the changes above, problems related to babel were observed when running `yarn start`. It has been known for some time that having the root babel config's dependencies spec'd in `packages/utils/collective/package.json` (which is the package that actually drives the babel cli) could lead to problems related to deduping, but such problems hadn't been experienced until now. Move the dependencies relevant to the root `babel.config.js` into the root `package.json` and update the explanatory comment in the config.
2020-01-29 22:19:25 +00:00
"@babel/cli": "7.7.4",
"@babel/core": "7.7.4",
"@babel/plugin-proposal-class-properties": "7.7.4",
"@babel/plugin-proposal-decorators": "7.7.4",
"@babel/plugin-proposal-export-default-from": "7.7.4",
"@babel/plugin-proposal-export-namespace-from": "7.7.4",
"@babel/plugin-proposal-nullish-coalescing-operator": "7.7.4",
"@babel/plugin-proposal-optional-chaining": "7.7.4",
"@babel/plugin-syntax-dynamic-import": "7.7.4",
"@babel/plugin-transform-runtime": "7.7.4",
"@babel/preset-env": "7.7.4",
"@babel/preset-typescript": "7.7.4",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"@typescript-eslint/parser": "2.20.0",
refactor(@embark/library-manager): restrict versionable packages to only solc BREAKING CHANGE: Remove support for specifying the versions of `web3` and `ipfs-api` in a project's `embark.json`.
2019-11-07 16:28:09 +00:00
"babel-eslint": "10.0.3",
build(deps-dev): update other @storybook packages to 5.3.9 This follows on PR #2227. Satisfy new peer dependencies following the bumps: `@storybook/core`, `regenerator-runtime`. Follow storybook's instructions to add `@storybook/preset-create-react-app` as a dependency and also add the needed `.storybook/main.js` in `packages/cockpit/ui/`. Storybook is sensitive to presets being in the immediate project's `node_modules` so add a `"nohoist"` in the root `package.json` for `"embark-ui/@storybook/**"`. After making the changes above, problems related to babel were observed when running `yarn start`. It has been known for some time that having the root babel config's dependencies spec'd in `packages/utils/collective/package.json` (which is the package that actually drives the babel cli) could lead to problems related to deduping, but such problems hadn't been experienced until now. Move the dependencies relevant to the root `babel.config.js` into the root `package.json` and update the explanatory comment in the config.
2020-01-29 22:19:25 +00:00
"babel-plugin-dynamic-import-node": "2.3.0",
"babel-plugin-macros": "2.7.1",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"chalk": "2.4.2",
build(deps-dev): bump coveralls from 3.0.6 to 3.0.9 Bumps [coveralls](https://github.com/nickmerwin/node-coveralls) from 3.0.6 to 3.0.9. - [Release notes](https://github.com/nickmerwin/node-coveralls/releases) - [Commits](https://github.com/nickmerwin/node-coveralls/commits/3.0.9) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-11-28 07:08:25 +00:00
"coveralls": "3.0.9",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"eslint": "6.8.0",
"eslint-plugin-import": "2.18.2",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"find-up": "4.1.0",
fix: type checker and linter errors on master after recent PR merges
2019-09-04 23:05:53 +00:00
"form-data": "2.5.1",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"fs-extra": "8.1.0",
build: implement collective typecheck This PR replaces #2057. Implement a collective typecheck action that can be invoked in the root of the monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`. Include the watch-mode typecheck action as part of `yarn start` (a.k.a `yarn watch`). To activate collective typecheck for a package in the monorepo, its `package.json` file should specify: ``` { "embark-collective": { "typecheck": true } } ``` *-or-* ``` { "embark-collective": { "typecheck": {...} } } ``` Where `{...}` above is a `tsconfig.json` fragment that will be merged into the config generated for the package according the same rules that `tsc` applies when merging [configs][config]. When collective typecheck begins, it generates a `tsconfig.json` for the root of the monorepo and for each package that is activated for the action. If the generated JSON is different than what's on disk for the respective root/package config, or if the config is not present on disk, then it will be written. Changes to generated `tsconfig.json` files should be committed; such changes will arise when there are structural changes to the monorepo, e.g. a package is added, removed, moved and/or the directory containing it is renamed. Since the configs are only generated at the beginning of collective typecheck, when structural changes are made in the monorepo `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) should be restarted. Nearly all of the packages in the monorepo (i.e. all those for which it makes sense) have been activated for collective typecheck. Even those packages that don't contain `.ts` sources are activated because `tsc` can make better sense of the code base as a whole owing to the project references included in the generated `tsconfig.json` files. Also, owing to the fully cross-referenced `tsconfig.json` files, it's possible for `tsc` to type check the whole code base without babel (`yarn build` or `yarn watch:build`) having been run beforehand. **NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive: on this author's 2019 MacBook Pro it takes around three minutes, the fans spin up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has completed, the next full typecheck will complete in a few seconds or less; and when running in watch-mode there is likewise a *big* speedup once a full typecheck has completed, whether that full check happened before it's running in watch-mode or when watch-mode itself resulted in a full check before switching automatically to incremental check, as well a corresponding *big* reduction in resource consumption. A full check will be needed any time `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`. The combination of options in each generated package-level `tsconfig.json` and the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript declaration files) into the `dist/` directory of each package. That output is intended to live side-by-side with babel's output, and therefore the `"rootDir"` option in each generated config is set to `"./src"`. In projects activated for collective typecheck, `.js` may be converted to `.ts` and/or `.ts` sources may be added without any additional changes needed in package-level `package.json`. --- Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into `packages/core/core` (`embark-core`), refactor other packages' imports accordingly, and delete `packages/core/typings` from the monorepo. This results in some similarly named but incompatible types exported from `embark-core` (e.g. `Events` and `EmbarkEvents`, the latter being the one from `packages/core/typings`); future refactoring should consolidate those types. To avoid circular dependency relationships it's also necessary to split out `Engine` from `embark-core` into its own package (`embark-engine`) and to introduce a bit of duplication, e.g. the `Maybe` type that's now defined in both `embark-i18n` and `embark-core`. In the process of the types reorg, move many dependencies spec'd in various `package.json` to the `package.json` of the package/s that actually depend on them, e.g. many are moved from `packages/embark/package.json` to `packages/core/engine/package.json`. Related to those moves, fix some Node.js `require`-logic related to bug-prone dependency resolution. Fix all type errors that appeared as a result of activating collective typecheck across the whole monorepo. Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors. Tidy up and add a few items in the root `package.json` scripts. Bump lerna from `3.16.4` to `3.19.0`. Bumpt typescript from `3.6.3` to `3.7.2`. Bumpt tslint from `5.16.0` to `5.20.1`. Make various changes related to packages' `import`/`require`ing packages that weren't spec'd in their respective `package.json`. More refactoring is needed in this regard, but changes were made as the problems were observed in the process of authoring this PR. [config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
"lerna": "3.19.0",
build(deps-dev): update other @storybook packages to 5.3.9 This follows on PR #2227. Satisfy new peer dependencies following the bumps: `@storybook/core`, `regenerator-runtime`. Follow storybook's instructions to add `@storybook/preset-create-react-app` as a dependency and also add the needed `.storybook/main.js` in `packages/cockpit/ui/`. Storybook is sensitive to presets being in the immediate project's `node_modules` so add a `"nohoist"` in the root `package.json` for `"embark-ui/@storybook/**"`. After making the changes above, problems related to babel were observed when running `yarn start`. It has been known for some time that having the root babel config's dependencies spec'd in `packages/utils/collective/package.json` (which is the package that actually drives the babel cli) could lead to problems related to deduping, but such problems hadn't been experienced until now. Move the dependencies relevant to the root `babel.config.js` into the root `package.json` and update the explanatory comment in the config.
2020-01-29 22:19:25 +00:00
"lodash.clonedeep": "4.5.0",
chore: introduce release script This commit automates our release process. It takes care of a couple of things: - Bumps version number in package.json as needed, see below - Generates changelog based on commit history - Create release commit - Tags release commit - Pushes release commit including tags to upstream repository - Publishes release on npm There are a couple of options supported. The base command is run like this: ``` npm run release ``` This will do a minor bump and try to push to `origin` on `master`. However, this can be altered with the following options. ``` npm run release -- --dry-run ``` Can be used to perform dry run and not actually committing, tagging, pushing, publishing anything. ``` npm run release -- --release-as <something> ``` Something can be `minor`, `major`, `patch` or anything you want `foo`, `1.0.0` etc. ``` npm run release -- --prerelease alpha ``` Will create a prerelease version a la `4.0.0-alpha.x`. ``` npm run release -- --prerelease alpha --npm-dist-tag next ``` Publishes a dist tag on npm using dist tag `next` ``` npm run release -- --sign ``` Signs the release commit (you need to have PGP setup for that). ``` npm run release -- --repo-origin pascal --repo-branch foo/bar ``` Pushes the release commit into `pascal/foo/bar`.
2018-11-09 13:26:08 +00:00
"minimist": "1.2.0",
build: bump npm-run-all to the latest version
2018-11-26 18:21:35 +00:00
"npm-run-all": "4.1.5",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"nyc": "13.1.0",
ci: implement a nightlies GitHub Actions workflow Implement a GitHub Actions workflow in `.github/workflows/nightlies.yml` named *Nightlies*, which is scheduled to run once daily at 00:00 UTC. At present the workflow includes one job named *release*, which is responsible for publishing prerelease GitHub releases and NPM packages. Each prerelease created (per package) will have a `nightly` [semver identifier][preid], and each successive nightly release will be paired with the `nightly` [dist-tag][dist-tag] on the NPM registry (per package). During the release job, actions taken in this GitHub repository (commits, tags, releases) and on the NPM registry (package publication) will be performed using credentials associated with the following accounts: * https://github.com/embarkbot * https://www.npmjs.com/~embarkbot For that purpose, corresponding [secrets][secrets] (link requires admin access) were created in this repository consisting of API tokens generated for the @embarkbot GitHub and NPM accounts. Logins for the @embarkbot accounts themselves are protected by 2FA. Implement `scripts/nightly-release.js` (`npm run release:nightly`), which is responsible for running `lerna publish` in the GitHub Actions workflow. Also implement `scripts/stable-release.js` (`npm run release:stable`), which is intended to be run locally by someone on the Embark Team. Both scripts borrow heavily from the existing `scripts/release.js`, and the process of authoring and experimenting with them influenced refactors to the latter. Use a `--force-publish` major-release strategy to prevent major-version drift between packages in the monorepo. How it works: when the stable-release script is run (`npm run release:stable`), if the current prerelease version involves a major version increase relative to the most recent stable release then **all** packages are bumped to the new major stable version. Otherwise, only the packages currently in prerelease are graduated to the new minor/patch stable version. In either case, the `nightly` dist-tag of each package published is updated to resolve to the new stable version. The reason for adopting this strategy *(a decision which can be revisited and changed any time in the future)* is based on a concern that downstream users would have a confusing developer UX if across `embark-*` packages there are differing major versions. To understand how the major-version drift would happen, consider the following hypothetical scenario where `--force-publish` *isn't* used in stable releases and `nightly` dist-tags aren't updated to resolve to the latest stable version: assume the current stable version is `6.5.4`. A breaking change lands for `embark-core`. The next nightly release bumps `embark-core` and about 40 other packages to `7.0.0-nightly.0`. However, `embark-utils` (and others) isn't bumped because it doesn't depend on `embark-core`. Later, without any intervening changes to `embark-utils`, the prerelease is graduated so that `embark-core`, etc. bump to `7.0.0`. So then some `embark-*` packages are at major version `7` while others are still at `6`. *Note* that this is the case even though this monorepo uses Lerna's *"fixed"* versioning mode. Inside the monorepo, `lerna` makes sure that everything is okay, i.e. with respect to automatically updating dependents' version specifiers for their dependencies that are within the monorepo. But for downstream users things are a bit more complex. If someone wanted to use `embark-utils` on its own and specified `^7.0.0` as the version range (after observing that `embark` itself is in a `7.x` series) it won't work because `embark-utils` is still in `6.x`. In the general case, users may have to manually cross-check major versions of various `embark-*` packages that they specify in their projects' `package.json` files. There are tools like [npm-check-updates][ncu] that can make the task easier, but there is still likely to be some confusion, especially given the large and growing number of packages in this monorepo. Another area of confusion would exist around the `nightly` dist-tag. In the scenario above, `embark-core@nightly` (and/or `@nightly` of its dependents, e.g. `embark`) would resolve to `7.0.0-nightly.0` but `embark-utils@nightly` would resolve to some `6.5.4-nightly.N` (:bomb:), i.e. a prerelease version that predates the current stable `6.5.4` release of `embark-utils` (and *might* not include all changes that landed in `embark-utils` prior to that stable release). By bumping all packages each time there is a major stable release, and by having the `nightly` dist-tag always point to a package's most recent release (whether stable or prerelease), the problems described above are avoided. To see the `--force-publish` major-release strategy in action take a look at the [commit history][history] for the [nightly-release-workflow-tester][mb-nrwt] repo together with the *Versions* tab of the NPM pages for the [foo][foo], [bar][bar], [baz][baz], and [quux][quux] packages. Ignore the version history for `<= 2.0.1` because those pre/releases were made with a different strategy than the current one. Refactor the existing `scripts/release.js` to make it more flexible generally and with respect to options that can be forwarded to `lerna`. In particular, it's now possible to run `lerna version` instead of `lerna publish` (the default behavior) by using the `--version-only` cli option; when combining that usage with `--skip-qa` and `--no-push` it's possible to conveniently and quickly experiment with the [`bump` positional][bump] and additional options such as `--force-publish`, `--conventional-prerelease`, and `--conventional-graduate`, i.e. to better understand how `lerna` will update package versions. That ability should make it much simpler to figure out the best course of action to take locally (manually) when a nightly release completely or partially failed (which could happen for a number of reasons), as well for other scenarios such as making a minor/patch release in a previous line of major releases, or when making two/more successive stable releases without a nightly release having happened in the meantime. An important change to `scripts/release.js` is that by default it makes use of the `--create-release github` option for `lerna version|publish`. For that to work, an environment variable named `GH_TOKEN` must be defined with a properly [scoped][scopes] GitHub [personal access token][pa-token] (`public_repo` scope is sufficient for creating releases). The same is true for `scripts/stable-release.js`. Delete the `.github/PULL_REQUEST_TEMPLATE` directory and the templates it contained. Unlike for GitHub issue creation, there is no prompt-page for picking from a repo's PR templates; to use a PR template a `template=[name]` [query parameter][template-query] must be appended to the URL of the PR creation page. So the PR templates ended up unused by the Embark Team and external contributors because it's not convenient to use them. Restore the default PR template we had in place some time ago (with some small revisions) since it seems like a helpful starting point, especially for external contributors. Consistently use all-lowercase filenames for ISSUE/PR templates. [preid]: https://semver.org/#spec-item-9 [dist-tag]: https://docs.npmjs.com/cli/dist-tag [secrets]: https://github.com/embarklabs/embark/settings/secrets [ncu]: https://www.npmjs.com/package/npm-check-updates [history]: https://github.com/michaelsbradleyjr/nightly-release-workflow-tester/commits/master [mb-nrwt]: https://github.com/michaelsbradleyjr/nightly-release-workflow-tester/ [foo]: https://www.npmjs.com/package/nightly-release-workflow-tester-foo?activeTab=versions [bar]: https://www.npmjs.com/package/nightly-release-workflow-tester-bar?activeTab=versions [baz]: https://www.npmjs.com/package/nightly-release-workflow-tester-baz?activeTab=versions [quux]: https://www.npmjs.com/package/nightly-release-workflow-tester-quux?activeTab=versions [bump]: https://github.com/lerna/lerna/tree/master/commands/version#semver-bump [scopes]: https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/ [pa-token]: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line [template-query]: https://help.github.com/en/github/building-a-strong-community/creating-a-pull-request-template-for-your-repository
2020-01-07 17:09:38 +00:00
"rimraf": "3.0.0",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"semver": "5.6.0",
"typescript": "3.7.2"
spec node, npm in "engines" of package.json
2018-09-18 21:28:50 +00:00
},
"engines": {
feat: support Node.js v12.x and newer Remove the `<12.0.0` restriction re: Node.js version in the `"engines"` settings for all the packages in the monorepo that had that restriction. Add missing `"engines"` settings in `packages/plugins/snark/package.json`. Adjust the Azure Pipelines config to include builds for Node.js v12.x and v13.x. Bump `solc` to `0.4.26` in `dapps/tests/app` and `dapps/tests/contracts`. It was discovered that older versions suffered a fatal `Maximum call stack size exceeded` error when run on Windows with Node.js v12.x or newer. Display a warning re: the bad combo (solc version + Windows + Node version) if it's detected at runtime. Adjust the root `yarn.lock` so that the `sha3` transitive dependency resolves to a newer version that is compatible with Node v13.x.
2020-01-17 17:32:32 +00:00
"node": ">=10.17.0",
build: bump all packages' engines settings (#1985) BREAKING CHANGE: node: >=10.17.0 <12.0.0 npm: >=6.11.3 yarn: >=1.19.1 node v10.17.0 is the latest in the 10.x series and is still in the Active LTS lifecycle. Embark is still not compatible with node's 12.x and 13.x series (because of some dependencies), otherwise it would probably make sense to bump our minimum supported node version all the way to the most recent 12.x release. npm v6.11.3 is the version that's bundled with node v10.17.0. yarn v1.19.1 is the most recent version as of the time node v10.17.0 was released.
2019-10-24 13:52:05 +00:00
"npm": ">=6.11.3",
"yarn": ">=1.19.1"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
},
build: tidy up the monorepo in prep for v5.0.0-alpha.0
2019-09-27 17:58:08 +00:00
"license": "MIT",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"name": "root",
"private": true,
"scripts": {
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"build": "node scripts/monorun --stream build",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"build:no-ui": "npm run build -- --ignore embark-ui",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"ci": "node scripts/monorun --ignore embark-dapp-* --stream ci",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"ci:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream ci",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"ci:full": "npm-run-all cylock cwtree typecheck lint build:no-ui \"test -- --concurrency={1}\" ci:dapps cwtree -- 1",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"clean": "node scripts/monorun --stream clean",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"clean:full": "npx npm-run-all clean clean:top",
build: implement collective typecheck This PR replaces #2057. Implement a collective typecheck action that can be invoked in the root of the monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`. Include the watch-mode typecheck action as part of `yarn start` (a.k.a `yarn watch`). To activate collective typecheck for a package in the monorepo, its `package.json` file should specify: ``` { "embark-collective": { "typecheck": true } } ``` *-or-* ``` { "embark-collective": { "typecheck": {...} } } ``` Where `{...}` above is a `tsconfig.json` fragment that will be merged into the config generated for the package according the same rules that `tsc` applies when merging [configs][config]. When collective typecheck begins, it generates a `tsconfig.json` for the root of the monorepo and for each package that is activated for the action. If the generated JSON is different than what's on disk for the respective root/package config, or if the config is not present on disk, then it will be written. Changes to generated `tsconfig.json` files should be committed; such changes will arise when there are structural changes to the monorepo, e.g. a package is added, removed, moved and/or the directory containing it is renamed. Since the configs are only generated at the beginning of collective typecheck, when structural changes are made in the monorepo `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) should be restarted. Nearly all of the packages in the monorepo (i.e. all those for which it makes sense) have been activated for collective typecheck. Even those packages that don't contain `.ts` sources are activated because `tsc` can make better sense of the code base as a whole owing to the project references included in the generated `tsconfig.json` files. Also, owing to the fully cross-referenced `tsconfig.json` files, it's possible for `tsc` to type check the whole code base without babel (`yarn build` or `yarn watch:build`) having been run beforehand. **NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive: on this author's 2019 MacBook Pro it takes around three minutes, the fans spin up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has completed, the next full typecheck will complete in a few seconds or less; and when running in watch-mode there is likewise a *big* speedup once a full typecheck has completed, whether that full check happened before it's running in watch-mode or when watch-mode itself resulted in a full check before switching automatically to incremental check, as well a corresponding *big* reduction in resource consumption. A full check will be needed any time `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`. The combination of options in each generated package-level `tsconfig.json` and the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript declaration files) into the `dist/` directory of each package. That output is intended to live side-by-side with babel's output, and therefore the `"rootDir"` option in each generated config is set to `"./src"`. In projects activated for collective typecheck, `.js` may be converted to `.ts` and/or `.ts` sources may be added without any additional changes needed in package-level `package.json`. --- Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into `packages/core/core` (`embark-core`), refactor other packages' imports accordingly, and delete `packages/core/typings` from the monorepo. This results in some similarly named but incompatible types exported from `embark-core` (e.g. `Events` and `EmbarkEvents`, the latter being the one from `packages/core/typings`); future refactoring should consolidate those types. To avoid circular dependency relationships it's also necessary to split out `Engine` from `embark-core` into its own package (`embark-engine`) and to introduce a bit of duplication, e.g. the `Maybe` type that's now defined in both `embark-i18n` and `embark-core`. In the process of the types reorg, move many dependencies spec'd in various `package.json` to the `package.json` of the package/s that actually depend on them, e.g. many are moved from `packages/embark/package.json` to `packages/core/engine/package.json`. Related to those moves, fix some Node.js `require`-logic related to bug-prone dependency resolution. Fix all type errors that appeared as a result of activating collective typecheck across the whole monorepo. Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors. Tidy up and add a few items in the root `package.json` scripts. Bump lerna from `3.16.4` to `3.19.0`. Bumpt typescript from `3.6.3` to `3.7.2`. Bumpt tslint from `5.16.0` to `5.20.1`. Make various changes related to packages' `import`/`require`ing packages that weren't spec'd in their respective `package.json`. More refactoring is needed in this regard, but changes were made as the problems were observed in the process of authoring this PR. [config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
"clean:top": "npm run reset:top && npx rimraf node_modules",
test: collect code coverage in the root of the monorepo Implement scripts to collect coverage reports (JSON format) from all packages in the monorepo that generate such reports. Reports are copied to `<root>/.nyc_output/coverage-[pkg-dir-name].json`. Implement scripts to generate a combined html report in `<root>/coverage`. Adjust root `reset` and `clean` scripts to delete `<root>/.nyc_output` and `<root>/coverage`. Implement a script in `<root>/package.json` to generate a `text-lcov` report and upload it to coveralls from CI builds. Remove coveralls from `packages/embark`. Supply `packages/embark` with an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Use nyc with `embarkjs`'s test suite: supply an nyc configuration in its `package.json` and have its `"test":` script generate both `json` and `html` reports. Adjust `embarkjs`'s tests for more accurate coverage reporting.
2019-04-28 21:33:41 +00:00
"coverage": "npm-run-all coverage:collect coverage:report",
"coverage:collect": "node scripts/coverage-collect",
"coverage:coveralls": "nyc report --reporter=text-lcov | coveralls || exit 0",
"coverage:report": "node scripts/coverage-report",
"coveralls": "npm-run-all coverage:collect coverage:coveralls",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"cwtree": "node scripts/check-working-tree",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"cylock": "node scripts/check-yarn-lock",
chore: move embark site into mono repo
2019-04-18 10:33:52 +00:00
"deploy:site": "node site/deploy-site",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"globalize": "node scripts/globalize",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"lint": "npm-run-all lint:*",
"lint:packages": "node scripts/monorun --parallel lint",
"lint:top": "eslint *.js scripts/",
build: implement collective typecheck This PR replaces #2057. Implement a collective typecheck action that can be invoked in the root of the monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`. Include the watch-mode typecheck action as part of `yarn start` (a.k.a `yarn watch`). To activate collective typecheck for a package in the monorepo, its `package.json` file should specify: ``` { "embark-collective": { "typecheck": true } } ``` *-or-* ``` { "embark-collective": { "typecheck": {...} } } ``` Where `{...}` above is a `tsconfig.json` fragment that will be merged into the config generated for the package according the same rules that `tsc` applies when merging [configs][config]. When collective typecheck begins, it generates a `tsconfig.json` for the root of the monorepo and for each package that is activated for the action. If the generated JSON is different than what's on disk for the respective root/package config, or if the config is not present on disk, then it will be written. Changes to generated `tsconfig.json` files should be committed; such changes will arise when there are structural changes to the monorepo, e.g. a package is added, removed, moved and/or the directory containing it is renamed. Since the configs are only generated at the beginning of collective typecheck, when structural changes are made in the monorepo `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) should be restarted. Nearly all of the packages in the monorepo (i.e. all those for which it makes sense) have been activated for collective typecheck. Even those packages that don't contain `.ts` sources are activated because `tsc` can make better sense of the code base as a whole owing to the project references included in the generated `tsconfig.json` files. Also, owing to the fully cross-referenced `tsconfig.json` files, it's possible for `tsc` to type check the whole code base without babel (`yarn build` or `yarn watch:build`) having been run beforehand. **NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive: on this author's 2019 MacBook Pro it takes around three minutes, the fans spin up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has completed, the next full typecheck will complete in a few seconds or less; and when running in watch-mode there is likewise a *big* speedup once a full typecheck has completed, whether that full check happened before it's running in watch-mode or when watch-mode itself resulted in a full check before switching automatically to incremental check, as well a corresponding *big* reduction in resource consumption. A full check will be needed any time `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`. The combination of options in each generated package-level `tsconfig.json` and the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript declaration files) into the `dist/` directory of each package. That output is intended to live side-by-side with babel's output, and therefore the `"rootDir"` option in each generated config is set to `"./src"`. In projects activated for collective typecheck, `.js` may be converted to `.ts` and/or `.ts` sources may be added without any additional changes needed in package-level `package.json`. --- Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into `packages/core/core` (`embark-core`), refactor other packages' imports accordingly, and delete `packages/core/typings` from the monorepo. This results in some similarly named but incompatible types exported from `embark-core` (e.g. `Events` and `EmbarkEvents`, the latter being the one from `packages/core/typings`); future refactoring should consolidate those types. To avoid circular dependency relationships it's also necessary to split out `Engine` from `embark-core` into its own package (`embark-engine`) and to introduce a bit of duplication, e.g. the `Maybe` type that's now defined in both `embark-i18n` and `embark-core`. In the process of the types reorg, move many dependencies spec'd in various `package.json` to the `package.json` of the package/s that actually depend on them, e.g. many are moved from `packages/embark/package.json` to `packages/core/engine/package.json`. Related to those moves, fix some Node.js `require`-logic related to bug-prone dependency resolution. Fix all type errors that appeared as a result of activating collective typecheck across the whole monorepo. Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors. Tidy up and add a few items in the root `package.json` scripts. Bump lerna from `3.16.4` to `3.19.0`. Bumpt typescript from `3.6.3` to `3.7.2`. Bumpt tslint from `5.16.0` to `5.20.1`. Make various changes related to packages' `import`/`require`ing packages that weren't spec'd in their respective `package.json`. More refactoring is needed in this regard, but changes were made as the problems were observed in the process of authoring this PR. [config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
"package": "lerna exec --concurrency=1 --no-private --stream -- npm pack",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"postclean": "npx lerna clean --yes",
"postreboot": "yarn install",
"postreboot:full": "yarn install",
"preci:full": "yarn install",
"preqa:full": "yarn install",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"qa": "node scripts/monorun --ignore embark-dapp-* --stream qa",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"qa:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream qa",
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"qa:full": "npm-run-all cylock cwtree reboot:full cwtree typecheck lint build \"test -- --concurrency={1}\" qa:dapps cwtree -- 1",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"reboot": "npm run clean",
"reboot:full": "npm run clean:full",
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"release": "node scripts/release",
ci: implement a nightlies GitHub Actions workflow Implement a GitHub Actions workflow in `.github/workflows/nightlies.yml` named *Nightlies*, which is scheduled to run once daily at 00:00 UTC. At present the workflow includes one job named *release*, which is responsible for publishing prerelease GitHub releases and NPM packages. Each prerelease created (per package) will have a `nightly` [semver identifier][preid], and each successive nightly release will be paired with the `nightly` [dist-tag][dist-tag] on the NPM registry (per package). During the release job, actions taken in this GitHub repository (commits, tags, releases) and on the NPM registry (package publication) will be performed using credentials associated with the following accounts: * https://github.com/embarkbot * https://www.npmjs.com/~embarkbot For that purpose, corresponding [secrets][secrets] (link requires admin access) were created in this repository consisting of API tokens generated for the @embarkbot GitHub and NPM accounts. Logins for the @embarkbot accounts themselves are protected by 2FA. Implement `scripts/nightly-release.js` (`npm run release:nightly`), which is responsible for running `lerna publish` in the GitHub Actions workflow. Also implement `scripts/stable-release.js` (`npm run release:stable`), which is intended to be run locally by someone on the Embark Team. Both scripts borrow heavily from the existing `scripts/release.js`, and the process of authoring and experimenting with them influenced refactors to the latter. Use a `--force-publish` major-release strategy to prevent major-version drift between packages in the monorepo. How it works: when the stable-release script is run (`npm run release:stable`), if the current prerelease version involves a major version increase relative to the most recent stable release then **all** packages are bumped to the new major stable version. Otherwise, only the packages currently in prerelease are graduated to the new minor/patch stable version. In either case, the `nightly` dist-tag of each package published is updated to resolve to the new stable version. The reason for adopting this strategy *(a decision which can be revisited and changed any time in the future)* is based on a concern that downstream users would have a confusing developer UX if across `embark-*` packages there are differing major versions. To understand how the major-version drift would happen, consider the following hypothetical scenario where `--force-publish` *isn't* used in stable releases and `nightly` dist-tags aren't updated to resolve to the latest stable version: assume the current stable version is `6.5.4`. A breaking change lands for `embark-core`. The next nightly release bumps `embark-core` and about 40 other packages to `7.0.0-nightly.0`. However, `embark-utils` (and others) isn't bumped because it doesn't depend on `embark-core`. Later, without any intervening changes to `embark-utils`, the prerelease is graduated so that `embark-core`, etc. bump to `7.0.0`. So then some `embark-*` packages are at major version `7` while others are still at `6`. *Note* that this is the case even though this monorepo uses Lerna's *"fixed"* versioning mode. Inside the monorepo, `lerna` makes sure that everything is okay, i.e. with respect to automatically updating dependents' version specifiers for their dependencies that are within the monorepo. But for downstream users things are a bit more complex. If someone wanted to use `embark-utils` on its own and specified `^7.0.0` as the version range (after observing that `embark` itself is in a `7.x` series) it won't work because `embark-utils` is still in `6.x`. In the general case, users may have to manually cross-check major versions of various `embark-*` packages that they specify in their projects' `package.json` files. There are tools like [npm-check-updates][ncu] that can make the task easier, but there is still likely to be some confusion, especially given the large and growing number of packages in this monorepo. Another area of confusion would exist around the `nightly` dist-tag. In the scenario above, `embark-core@nightly` (and/or `@nightly` of its dependents, e.g. `embark`) would resolve to `7.0.0-nightly.0` but `embark-utils@nightly` would resolve to some `6.5.4-nightly.N` (:bomb:), i.e. a prerelease version that predates the current stable `6.5.4` release of `embark-utils` (and *might* not include all changes that landed in `embark-utils` prior to that stable release). By bumping all packages each time there is a major stable release, and by having the `nightly` dist-tag always point to a package's most recent release (whether stable or prerelease), the problems described above are avoided. To see the `--force-publish` major-release strategy in action take a look at the [commit history][history] for the [nightly-release-workflow-tester][mb-nrwt] repo together with the *Versions* tab of the NPM pages for the [foo][foo], [bar][bar], [baz][baz], and [quux][quux] packages. Ignore the version history for `<= 2.0.1` because those pre/releases were made with a different strategy than the current one. Refactor the existing `scripts/release.js` to make it more flexible generally and with respect to options that can be forwarded to `lerna`. In particular, it's now possible to run `lerna version` instead of `lerna publish` (the default behavior) by using the `--version-only` cli option; when combining that usage with `--skip-qa` and `--no-push` it's possible to conveniently and quickly experiment with the [`bump` positional][bump] and additional options such as `--force-publish`, `--conventional-prerelease`, and `--conventional-graduate`, i.e. to better understand how `lerna` will update package versions. That ability should make it much simpler to figure out the best course of action to take locally (manually) when a nightly release completely or partially failed (which could happen for a number of reasons), as well for other scenarios such as making a minor/patch release in a previous line of major releases, or when making two/more successive stable releases without a nightly release having happened in the meantime. An important change to `scripts/release.js` is that by default it makes use of the `--create-release github` option for `lerna version|publish`. For that to work, an environment variable named `GH_TOKEN` must be defined with a properly [scoped][scopes] GitHub [personal access token][pa-token] (`public_repo` scope is sufficient for creating releases). The same is true for `scripts/stable-release.js`. Delete the `.github/PULL_REQUEST_TEMPLATE` directory and the templates it contained. Unlike for GitHub issue creation, there is no prompt-page for picking from a repo's PR templates; to use a PR template a `template=[name]` [query parameter][template-query] must be appended to the URL of the PR creation page. So the PR templates ended up unused by the Embark Team and external contributors because it's not convenient to use them. Restore the default PR template we had in place some time ago (with some small revisions) since it seems like a helpful starting point, especially for external contributors. Consistently use all-lowercase filenames for ISSUE/PR templates. [preid]: https://semver.org/#spec-item-9 [dist-tag]: https://docs.npmjs.com/cli/dist-tag [secrets]: https://github.com/embarklabs/embark/settings/secrets [ncu]: https://www.npmjs.com/package/npm-check-updates [history]: https://github.com/michaelsbradleyjr/nightly-release-workflow-tester/commits/master [mb-nrwt]: https://github.com/michaelsbradleyjr/nightly-release-workflow-tester/ [foo]: https://www.npmjs.com/package/nightly-release-workflow-tester-foo?activeTab=versions [bar]: https://www.npmjs.com/package/nightly-release-workflow-tester-bar?activeTab=versions [baz]: https://www.npmjs.com/package/nightly-release-workflow-tester-baz?activeTab=versions [quux]: https://www.npmjs.com/package/nightly-release-workflow-tester-quux?activeTab=versions [bump]: https://github.com/lerna/lerna/tree/master/commands/version#semver-bump [scopes]: https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/ [pa-token]: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line [template-query]: https://help.github.com/en/github/building-a-strong-community/creating-a-pull-request-template-for-your-repository
2020-01-07 17:09:38 +00:00
"release:nightly": "node scripts/nightly-release",
"release:stable": "node scripts/stable-release",
build: implement collective typecheck This PR replaces #2057. Implement a collective typecheck action that can be invoked in the root of the monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`. Include the watch-mode typecheck action as part of `yarn start` (a.k.a `yarn watch`). To activate collective typecheck for a package in the monorepo, its `package.json` file should specify: ``` { "embark-collective": { "typecheck": true } } ``` *-or-* ``` { "embark-collective": { "typecheck": {...} } } ``` Where `{...}` above is a `tsconfig.json` fragment that will be merged into the config generated for the package according the same rules that `tsc` applies when merging [configs][config]. When collective typecheck begins, it generates a `tsconfig.json` for the root of the monorepo and for each package that is activated for the action. If the generated JSON is different than what's on disk for the respective root/package config, or if the config is not present on disk, then it will be written. Changes to generated `tsconfig.json` files should be committed; such changes will arise when there are structural changes to the monorepo, e.g. a package is added, removed, moved and/or the directory containing it is renamed. Since the configs are only generated at the beginning of collective typecheck, when structural changes are made in the monorepo `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) should be restarted. Nearly all of the packages in the monorepo (i.e. all those for which it makes sense) have been activated for collective typecheck. Even those packages that don't contain `.ts` sources are activated because `tsc` can make better sense of the code base as a whole owing to the project references included in the generated `tsconfig.json` files. Also, owing to the fully cross-referenced `tsconfig.json` files, it's possible for `tsc` to type check the whole code base without babel (`yarn build` or `yarn watch:build`) having been run beforehand. **NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive: on this author's 2019 MacBook Pro it takes around three minutes, the fans spin up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has completed, the next full typecheck will complete in a few seconds or less; and when running in watch-mode there is likewise a *big* speedup once a full typecheck has completed, whether that full check happened before it's running in watch-mode or when watch-mode itself resulted in a full check before switching automatically to incremental check, as well a corresponding *big* reduction in resource consumption. A full check will be needed any time `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`. The combination of options in each generated package-level `tsconfig.json` and the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript declaration files) into the `dist/` directory of each package. That output is intended to live side-by-side with babel's output, and therefore the `"rootDir"` option in each generated config is set to `"./src"`. In projects activated for collective typecheck, `.js` may be converted to `.ts` and/or `.ts` sources may be added without any additional changes needed in package-level `package.json`. --- Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into `packages/core/core` (`embark-core`), refactor other packages' imports accordingly, and delete `packages/core/typings` from the monorepo. This results in some similarly named but incompatible types exported from `embark-core` (e.g. `Events` and `EmbarkEvents`, the latter being the one from `packages/core/typings`); future refactoring should consolidate those types. To avoid circular dependency relationships it's also necessary to split out `Engine` from `embark-core` into its own package (`embark-engine`) and to introduce a bit of duplication, e.g. the `Maybe` type that's now defined in both `embark-i18n` and `embark-core`. In the process of the types reorg, move many dependencies spec'd in various `package.json` to the `package.json` of the package/s that actually depend on them, e.g. many are moved from `packages/embark/package.json` to `packages/core/engine/package.json`. Related to those moves, fix some Node.js `require`-logic related to bug-prone dependency resolution. Fix all type errors that appeared as a result of activating collective typecheck across the whole monorepo. Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors. Tidy up and add a few items in the root `package.json` scripts. Bump lerna from `3.16.4` to `3.19.0`. Bumpt typescript from `3.6.3` to `3.7.2`. Bumpt tslint from `5.16.0` to `5.20.1`. Make various changes related to packages' `import`/`require`ing packages that weren't spec'd in their respective `package.json`. More refactoring is needed in this regard, but changes were made as the problems were observed in the process of authoring this PR. [config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
"reset": "node scripts/monorun --stream reset && npm-run-all reset:*",
"reset:top": "npx rimraf .tsconfig.collective.json .nyc_output coverage",
"reset:tsbuildinfo": "npx lerna exec --parallel -- npx rimraf node_modules/.cache/tsc",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"start": "node scripts/monorun --parallel start",
"test": "node scripts/monorun --ignore embark-dapp-* --stream test",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"test:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream test",
"test:full": "npm-run-all test test:dapps",
build: implement collective typecheck This PR replaces #2057. Implement a collective typecheck action that can be invoked in the root of the monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`. Include the watch-mode typecheck action as part of `yarn start` (a.k.a `yarn watch`). To activate collective typecheck for a package in the monorepo, its `package.json` file should specify: ``` { "embark-collective": { "typecheck": true } } ``` *-or-* ``` { "embark-collective": { "typecheck": {...} } } ``` Where `{...}` above is a `tsconfig.json` fragment that will be merged into the config generated for the package according the same rules that `tsc` applies when merging [configs][config]. When collective typecheck begins, it generates a `tsconfig.json` for the root of the monorepo and for each package that is activated for the action. If the generated JSON is different than what's on disk for the respective root/package config, or if the config is not present on disk, then it will be written. Changes to generated `tsconfig.json` files should be committed; such changes will arise when there are structural changes to the monorepo, e.g. a package is added, removed, moved and/or the directory containing it is renamed. Since the configs are only generated at the beginning of collective typecheck, when structural changes are made in the monorepo `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) should be restarted. Nearly all of the packages in the monorepo (i.e. all those for which it makes sense) have been activated for collective typecheck. Even those packages that don't contain `.ts` sources are activated because `tsc` can make better sense of the code base as a whole owing to the project references included in the generated `tsconfig.json` files. Also, owing to the fully cross-referenced `tsconfig.json` files, it's possible for `tsc` to type check the whole code base without babel (`yarn build` or `yarn watch:build`) having been run beforehand. **NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive: on this author's 2019 MacBook Pro it takes around three minutes, the fans spin up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has completed, the next full typecheck will complete in a few seconds or less; and when running in watch-mode there is likewise a *big* speedup once a full typecheck has completed, whether that full check happened before it's running in watch-mode or when watch-mode itself resulted in a full check before switching automatically to incremental check, as well a corresponding *big* reduction in resource consumption. A full check will be needed any time `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`. The combination of options in each generated package-level `tsconfig.json` and the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript declaration files) into the `dist/` directory of each package. That output is intended to live side-by-side with babel's output, and therefore the `"rootDir"` option in each generated config is set to `"./src"`. In projects activated for collective typecheck, `.js` may be converted to `.ts` and/or `.ts` sources may be added without any additional changes needed in package-level `package.json`. --- Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into `packages/core/core` (`embark-core`), refactor other packages' imports accordingly, and delete `packages/core/typings` from the monorepo. This results in some similarly named but incompatible types exported from `embark-core` (e.g. `Events` and `EmbarkEvents`, the latter being the one from `packages/core/typings`); future refactoring should consolidate those types. To avoid circular dependency relationships it's also necessary to split out `Engine` from `embark-core` into its own package (`embark-engine`) and to introduce a bit of duplication, e.g. the `Maybe` type that's now defined in both `embark-i18n` and `embark-core`. In the process of the types reorg, move many dependencies spec'd in various `package.json` to the `package.json` of the package/s that actually depend on them, e.g. many are moved from `packages/embark/package.json` to `packages/core/engine/package.json`. Related to those moves, fix some Node.js `require`-logic related to bug-prone dependency resolution. Fix all type errors that appeared as a result of activating collective typecheck across the whole monorepo. Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors. Tidy up and add a few items in the root `package.json` scripts. Bump lerna from `3.16.4` to `3.19.0`. Bumpt typescript from `3.6.3` to `3.7.2`. Bumpt tslint from `5.16.0` to `5.20.1`. Make various changes related to packages' `import`/`require`ing packages that weren't spec'd in their respective `package.json`. More refactoring is needed in this regard, but changes were made as the problems were observed in the process of authoring this PR. [config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
"typecheck": "node scripts/monorun --stream typecheck",
"typecheck:clean": "npm run typecheck -- -- -- --clean",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"watch": "node scripts/monorun --parallel watch",
"watch:build": "node scripts/monorun --parallel watch:build",
"watch:lint": "node scripts/monorun --parallel watch:lint",
"watch:test": "node scripts/monorun --ignore embark-dapp-* --parallel watch:test",
"watch:typecheck": "node scripts/monorun --parallel watch:typecheck"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
},
fix: ensure that packages properly specify their dependencies Many packages in the monorepo did not specify all of their dependencies; they were effectively relying on resolution in the monorepo's root `node_modules`. In a production release of `embark` and `embark[js]-*` packages this can lead to broken packages. To fix the problem currently and to help prevent it from happening again, make use of the `eslint-plugin-import` package's `import/no-extraneous-dependencies` and `import/no-unresolved` rules. In the root `tslint.json` set `"no-implicit-dependencies": true`, wich is the tslint equivalent of `import/no-extraneous-dependencies`; there is no tslint equivalent for `import/no-unresolved`, but we will eventually replace tslint with an eslint configuration that checks both `.js` and `.ts` files. For `import/no-unresolved` to work in our monorepo setup, in most packages add an `index.js` that has: ```js module.exports = require('./dist'); // or './dist/lib' in some cases ``` And point `"main"` in `package.json` to `"./index.js"`. Despite what's indicated in npm's documentation for `package.json`, it's also necessary to add `"index.js"` to the `"files"` array. Make sure that all `.js` files that can and should be linted are in fact linted. For example, files in `packages/embark/src/cmd/` weren't being linted and many test suites weren't being linted. Bump all relevant packages to `eslint@6.8.0`. Fix all linter errors that arose after these changes. Implement a `check-yarn-lock` script that's run as part of `"ci:full"` and `"qa:full"`, and can manually be invoked via `yarn cylock` in the root of the monorepo. The script exits with error if any specifiers are found in `yarn.lock` for `embark[js][-*]` and/or `@embarklabs/*` (with a few exceptions, cf. `scripts/check-yarn-lock.js`).
2020-02-20 23:47:01 +00:00
"eslintConfig": {
"extends": "./.eslintrc.json"
},
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
"workspaces": {
"packages": [
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"dapps/templates/*",
"dapps/tests/*",
feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile (#1878) * feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile * style: keep root package.json sorted cleanly * style: formatting with prettier where print-width is 200 Per the suggestion of @iurimatias * WIP: PR review * WIP: PR review * WIP: refactor to check if file.path is within dappPath() or os.tmpdir() * WIP: PR review * WIP: use native async functions (and related JS lang constructs) instead of the async library * WIP: async.eachObject is no longer a dependency
2019-09-11 17:08:52 +00:00
"packages/*",
"packages/cockpit/*",
refactor(@embark/): move several packages into logical folders (#1877)
2019-09-06 21:39:51 +00:00
"packages/core/*",
move embarkjs packages & remove embark- prefix from some folders (#1879) * chore(@embark/): move embarkjs packages to their own folder * chore(@embark/): rename embark-ui folder to cockpit * chore(@embark/): rename packages already in logical folders to remove embark- prefix chore(@embark/): rename packages already in logical folders to remove embark- prefix update package.json files to use correct eslint config remove core/* from package.json workspaces
2019-09-06 22:26:08 +00:00
"packages/embarkjs/*",
feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile (#1878) * feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile * style: keep root package.json sorted cleanly * style: formatting with prettier where print-width is 200 Per the suggestion of @iurimatias * WIP: PR review * WIP: PR review * WIP: refactor to check if file.path is within dappPath() or os.tmpdir() * WIP: PR review * WIP: use native async functions (and related JS lang constructs) instead of the async library * WIP: async.eachObject is no longer a dependency
2019-09-11 17:08:52 +00:00
"packages/plugins/*",
build: phase 1 of introducing embark-collective
2019-09-18 19:30:41 +00:00
"packages/stack/*",
build: tidy up the monorepo in prep for v5.0.0-alpha.0
2019-09-27 17:58:08 +00:00
"packages/utils/*"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
],
"nohoist": [
build: add nohoist entry for bootstrap package relative to dapps/templates/demo The demo's `embark.json` relies on the bootstrap package being within the demo's own `node_modules`, but in the monorepo yarn was hoisting it up to the root `node_modules`.
2019-05-15 14:27:22 +00:00
"embark-dapp-template-demo/bootstrap",
build: make DApp templates member packages of the monorepo Previously, templates were in a subdirectory of `packages/embark`. Reorganize them so that they are member packages of the monorepo. This allows them to cleanly depend on other members of the monorepo, e.g. `embarkjs-connector-web3`. It is desirable for the templates, in the context of the monorepo, to specify embark as a dependency, to take advantage of `npx embark test` (and it's a "forward looking" setup re: how we plan to evolve embark). However, if embark were to specify the template packages as dependencies a circular relationship would be introduced, which is [unsupported by Lerna][circular]. Therefore, revise the template generator so that all templates are resolved / fetched at runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer "built-ins" *per se*. This change won't be apparent to embark's users, but it does mean that the template generator won't work (in a production install of embark) if it can't connect to the npm registry, i.e. when the user runs `embark demo` or `embark new [--simple]`. When embark is inside the monorepo, templates are resolved and copied from the yarn workspace rather than being fetched from the registry, which is convenient for development. Also, any template dependencies that are members of the monorepo are linked into the copied template's `node_modules` rather than being installed from the registry, again for convenience. During template generation, remove scripts and dependencies that pertain only to membership in the monorepo; for now, that involves removing embark as a dependency since we're not quite ready for that arrangement to be the default, i.e. outside of the monorepo. Refactor the root scripts so that more of them can consistently be used with Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that don't support filtering generally have a `:full` postfix. Flip `clean` and `reset` scripts at the root and in the member packages for consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have each package run its `reset` script when its `clean` script is invoked (and that's all for now), relying on `lerna clean` to delete packages' `node_modules` in view of how Lerna's topological sorting works. Lift the implementation of `embark reset` into a private package in `packages/embark-reset` and make it a bundled dependency of embark. Packages in `dapps/*` depend on `embark-reset` directly and make use of it with `npx embark-reset` (but only in monorepo context). This removes a "wart" where reboots could show errors when embark's sources aren't already built in `packages/embark/dist`. Users will not notice any difference since `embark reset` works as before, transparently making use of the `embark-reset` package. The only downside to having it be a bundled dependency of embark is that bundled deps have all of their `node_modules` included in the tarball built with `npm pack` (that's why having the templates as bundled dependencies of embark isn't a viable approach). However, `embark-reset` only has one dependency, `rimraf`, which is a tiny module, so the cost seems acceptable. As part of the reorganization, move `test_dapps` into `dapps/tests` and `packages/embark/templates` into `dapps/templates`. Keep the directory names short but revise the package names to facilitate simple filtering with `embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some redundant ignore listings. Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid conflicts that could arise over network ports. The `ci:full` and `qa:full` scripts use `--concurrency=1` in all scopes, for two reasons: resource limitations on Travis and AppVeyor result in slower runs with concurrency >1, and if something fails in those contexts it's easier to see what went wrong when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`. Bump the Lerna version. [circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
"embark-dapp-test-app/embark-dapp-test-service",
feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile (#1878) * feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile * style: keep root package.json sorted cleanly * style: formatting with prettier where print-width is 200 Per the suggestion of @iurimatias * WIP: PR review * WIP: PR review * WIP: refactor to check if file.path is within dappPath() or os.tmpdir() * WIP: PR review * WIP: use native async functions (and related JS lang constructs) instead of the async library * WIP: async.eachObject is no longer a dependency
2019-09-11 17:08:52 +00:00
"embark-dapp-test-app/zeppelin-solidity",
build(deps-dev): update other @storybook packages to 5.3.9 This follows on PR #2227. Satisfy new peer dependencies following the bumps: `@storybook/core`, `regenerator-runtime`. Follow storybook's instructions to add `@storybook/preset-create-react-app` as a dependency and also add the needed `.storybook/main.js` in `packages/cockpit/ui/`. Storybook is sensitive to presets being in the immediate project's `node_modules` so add a `"nohoist"` in the root `package.json` for `"embark-ui/@storybook/**"`. After making the changes above, problems related to babel were observed when running `yarn start`. It has been known for some time that having the root babel config's dependencies spec'd in `packages/utils/collective/package.json` (which is the package that actually drives the babel cli) could lead to problems related to deduping, but such problems hadn't been experienced until now. Move the dependencies relevant to the root `babel.config.js` into the root `package.json` and update the explanatory comment in the config.
2020-01-29 22:19:25 +00:00
"embark-ui/@storybook/**",
refactor(@cockpit/ui): use non-ejected create-react-app (#1910) Change components/TexEditor re: how monaco-editor is loaded, since difficulties with monaco-editor were the original motivation for "ejecting" the CRA setup. Specify "nohoist" for `embark-ui/react-scripts` and all its dependencies to avoid potential headaches when upgrading the react-scripts, i.e. upgrading to newer versions of create-react-app.
2019-09-17 15:32:45 +00:00
"embark-ui/react-scripts",
"embark-ui/react-scripts/**",
feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile (#1878) * feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile * style: keep root package.json sorted cleanly * style: formatting with prettier where print-width is 200 Per the suggestion of @iurimatias * WIP: PR review * WIP: PR review * WIP: refactor to check if file.path is within dappPath() or os.tmpdir() * WIP: PR review * WIP: use native async functions (and related JS lang constructs) instead of the async library * WIP: async.eachObject is no longer a dependency
2019-09-11 17:08:52 +00:00
"embark/embark-test-contract-0",
"embark/embark-test-contract-1"
build: implement a monorepo with Lerna TL;DR ===== `yarn install` in a fresh clone of the repo. `yarn reboot` when switching branches. When pulling in these changes, there may be untracked files at the root in all/some of: ``` .embark/ .nyc_output/ coverage/ dist/ embark-ui/ test_apps/ ``` They can be safely deleted since those paths are no longer in use at the root. Many of the scripts in the top-level `package.json` support Lerna's [filter options]. For example: `yarn build --scope embark` build only `packages/embark`. `yarn build --ignore embark-ui` build everything except `packages/embark-ui`. Scoping scripts will be more useful when there are more packages in the monorepo and, for example, `yarn start` doesn't need to be invoked for all of them while working on just a few of them simultaneously, e.g `embark` and `embarkjs`. It's also possible to `cd` into a particular package and run its scripts directly: ``` cd packages/embark && yarn watch ``` Hot Topics & Questions ====================== What should be done about the [README][embark-readme] for `packages/embark`? Should the top-level README be duplicated in that package? Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly `packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we want to use [Independent mode][independent]? Scripts ======= If a package doesn't have a matching script, `lerna run` skips it automatically. For example, `packages/embark-ui` doesn't have a `typecheck` script. `yarn build` ------------ Runs babel, webpack, etc. according to a package's `build` script. `yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`. `yarn ci` --------- Runs a series of scripts relevant in a CI context according to a package's `ci` script. For `packages/embark` that's `lint typecheck build test package`. Also runs the `ci` script of the embedded `test_dapps` monorepo. `yarn clean` ------------ Runs rimraf, etc. according to a package's `clean` script. `yarn globalize` ---------------- Makes the development embark available on the global PATH, either via symlink (Linux, macOS) or a shim script (Windows). `yarn lint` ----------- Runs eslint, etc. according to a package's `lint` script. `yarn package` -------------- Invokes `npm pack` according to a package's `package` script. `yarn qa` --------- Very similar to `ci`, runs a series of scripts according to a package's `qa` script. The big difference between `ci` and `qa` is that at the top-level `qa` first kicks off `reboot:full`. There is a `preqa` script ([invoked automatically][npm-scripts]), which is a bit of a wart. It makes sure that `embark reset` can be run successfully in `packages/embark/templates/*` when the `reboot` script invokes the `reset` script. The `qa` script is invoked by `yarn release` before the latter proceeds to invoke `lerna publish`. `yarn reboot` ------------- Invokes the `reset` script and then does `yarn install`. The `reboot:full` variant invokes `reset:full` and then does `yarn install`. `yarn release` -------------- Works in concert with [lerna publish], which will prompt to verify the version before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has been seen to occasionally misbehave when not exited cleanly (e.g. creating a tag when it shouldn't have). ``` yarn release [bump] [--options] ``` * `[bump]` see [`publish` positionals][pub-pos] and [`version` positionals][ver-pos]; an exact version can also be specified. * `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump will default to whatever identifier is currently in use. * `--dist-tag` registry distribution tag, defaults to `latest`. * `--message` commit message format, defaults to `chore(release): %v`. * `--sign` indicates that the git commit and tag should be signed; not signed by default. * `--release-branch` default is `master`; must match the current branch. * `--git-remote` default is `origin`. * `--registry` default is `https://registry.npmjs.org/` per the top-level [`lerna.json`][lerna-json]. To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at `4.0.0-beta.0`) could do: ``` yarn release prerelease --dist-tag next ``` For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a filesystem git remote can be used. Condensend instructions: ``` mkdir -p ~/temp/clones && cd ~/temp/clones git clone git@github.com:embark-framework/embark.git cd ~/repos/embark git remote add FAKEembark ~/temp/clones/embark ``` in another terminal: ``` npm i -g verdaccio && verdaccio ``` in the first terminal: ``` yarn release --git-remote FAKEembark --registry http://localhost:4873/ ``` `yarn reset` ------------ Invokes cleaning and resetting steps according to a package's `reset` script. The big difference between `clean` and `reset` is that `reset` is intended to delete a package's `node_modules`. The `reset:full` variant deletes the monorepo's top-level `node_modules` at the end. That shouldn't be necessary too often, e.g. in day-to-day work when switching branches, which is why there is `reboot` / `reset` vs. `reboot:full` / `reset:full`. Errors may be seen related to invocation of `embark reset` if embark is not built, but `reset` will still complete successfully. `yarn start` ------------ Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a package's `start` script. `yarn test` ----------- Run mocha, etc. according to a package's `test` script. The `test:full` variant runs a series of scripts: `lint typecheck test test_dapps`. `yarn test_dapps` ----------------- Runs the `test` script of the embedded `test_dapps` monorepo. The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts of the embedded `test_dapps` monorepo, respectively. `yarn typecheck` ---------------- Runs tsc, etc. according to a package's `typecheck` script. Notes ===== `npx` is used in some of the top-level and package scripts to ensure the scripts can run even if `node_modules` is missing. [`"nohoist"`][nohoist] specifies a couple of embark packages because [`restrictPath`][restrictpath] is interfering with access to modules that are located in a higher-up `node_modules`. All dependencies in `packages/embark-ui` have been made `devDependencies` since its production build is self-contained. `packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted to match the formatting that Lerna will use going forward (entries in the log haven't been modified). Lerna will generate a CHANGELOG at the top-level and in each package. Since we're transitioning to a monorepo, things may look a little wonky with respect to old entries in `packages/embark/CHANGELOG.md` and going forward we need to consider how scoping our commits corresponds to member-packages of the monorepo. In `packages/embark`, `test` invokes `scripts/test`, which starts a child process wherein `process.env.DAPP_PATH` is a temporary path that has all of `packages/embark/dist/test` copied into it, so that paths to test helpers/fixtures don't need to be prefixed with `dist/test/` and so that a `.embark` directory doesn't get written into `packages/embark`. The `"engines"` specified in top-level and packages' `package.json` reflect a node and npm pair that match (a source of confusion in the past). The pair was chosen according to the first post v5 npm that's bundled with node. A `"runtime"` key/object has been introduced in `packages/embark/package.json` which is used as the basis for specifying the minimum version of node that can be used to run embark, and that's what is checked by `bin/embark`. Some changes have been introduced, e.g. in `lib/core/config` and `lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that `process.env.DAPP_PATH` / `fs.dappPath()` are the same as `process.cwd()`. There are probably several++ places where that assumption is still in effect, and we should work to identify and correct them. `embark reset` now deletes `embarkArtifacts/` within a dapp root, and `embarkArtifacts/` is git-ignored. `lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to `NODE_PATH` so that embark's modules can be resolved as expected whether embark's `node_modules` have been deduped or are installed in npm's flat "global style". `checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and slightly modified to support dependencies that have been hoisted into a higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been disabled for that script to avoid more involved changes to it. `test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` / `qa` scripts `npm install` embark from freshly built tarballs of the packages in the outer monorepo and then use that installation to run `embark test` in the dapps. This should allow us to rapidly detect breakage related to auto-bumps in transitive dependencies. [filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options [embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md [fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default [independent]: https://github.com/lerna/lerna#independent-mode [npm-scripts]: https://docs.npmjs.com/misc/scripts [lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish [pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals [ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals [lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11 [verdaccio]: https://www.npmjs.com/package/verdaccio [nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55 [restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
]
restructure; move code from tasks into libs
2015-06-28 02:20:07 +00:00
}
add npm package
2015-05-24 12:38:56 +00:00
}