build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
{
|
2019-03-08 19:58:27 +00:00
|
|
|
"description": "Test DApp for integration testing purposes",
|
build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
"devDependencies": {
|
2019-10-24 21:41:28 +00:00
|
|
|
"@babel/code-frame": "7.5.5",
|
2019-08-20 01:24:14 +00:00
|
|
|
"bootstrap": "3.4.1",
|
2020-01-20 15:57:35 +00:00
|
|
|
"embark": "^5.1.0-nightly.1",
|
2020-01-07 18:14:37 +00:00
|
|
|
"embark-dapp-test-service": "^5.0.0",
|
2020-01-20 15:57:35 +00:00
|
|
|
"embark-i18n": "^5.1.0-nightly.1",
|
|
|
|
|
"embark-reset": "^5.1.0-nightly.1",
|
|
|
|
|
"embarkjs": "^5.1.0-nightly.1",
|
|
|
|
|
"embarkjs-ens": "^5.1.0-nightly.1",
|
|
|
|
|
"embarkjs-ipfs": "^5.1.0-nightly.1",
|
|
|
|
|
"embarkjs-swarm": "^5.1.0-nightly.1",
|
|
|
|
|
"embarkjs-web3": "^5.1.0-nightly.1",
|
|
|
|
|
"embarkjs-whisper": "^5.1.0-nightly.1",
|
2019-09-18 19:30:41 +00:00
|
|
|
"fs-extra": "8.1.0",
|
build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
"haml": "0.4.3",
|
2019-08-14 07:53:56 +00:00
|
|
|
"jquery": "3.4.0",
|
2019-11-25 07:43:53 +00:00
|
|
|
"react": "16.12.0",
|
build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
"react-bootstrap": "0.32.4",
|
2019-11-25 23:16:46 +00:00
|
|
|
"react-dom": "16.12.0",
|
build: implement collective typecheck
This PR replaces #2057.
Implement a collective typecheck action that can be invoked in the root of the
monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`.
Include the watch-mode typecheck action as part of `yarn start` (a.k.a
`yarn watch`).
To activate collective typecheck for a package in the monorepo, its
`package.json` file should specify:
```
{
"embark-collective": {
"typecheck": true
}
}
```
*-or-*
```
{
"embark-collective": {
"typecheck": {...}
}
}
```
Where `{...}` above is a `tsconfig.json` fragment that will be merged into the
config generated for the package according the same rules that `tsc` applies
when merging [configs][config].
When collective typecheck begins, it generates a `tsconfig.json` for the root
of the monorepo and for each package that is activated for the action. If the
generated JSON is different than what's on disk for the respective root/package
config, or if the config is not present on disk, then it will be
written. Changes to generated `tsconfig.json` files should be committed; such
changes will arise when there are structural changes to the monorepo, e.g. a
package is added, removed, moved and/or the directory containing it is
renamed. Since the configs are only generated at the beginning of collective
typecheck, when structural changes are made in the monorepo `yarn typecheck`
(or `yarn start` or `yarn watch:typecheck`) should be restarted.
Nearly all of the packages in the monorepo (i.e. all those for which it makes
sense) have been activated for collective typecheck. Even those packages that
don't contain `.ts` sources are activated because `tsc` can make better sense
of the code base as a whole owing to the project references included in the
generated `tsconfig.json` files. Also, owing to the fully cross-referenced
`tsconfig.json` files, it's possible for `tsc` to type check the whole code
base without babel (`yarn build` or `yarn watch:build`) having been run
beforehand.
**NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive:
on this author's 2019 MacBook Pro it takes around three minutes, the fans spin
up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has
completed, the next full typecheck will complete in a few seconds or less; and
when running in watch-mode there is likewise a *big* speedup once a full
typecheck has completed, whether that full check happened before it's running
in watch-mode or when watch-mode itself resulted in a full check before
switching automatically to incremental check, as well a corresponding *big*
reduction in resource consumption. A full check will be needed any time
`yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh
clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`.
The combination of options in each generated package-level `tsconfig.json` and
the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript
declaration files) into the `dist/` directory of each package. That
output is intended to live side-by-side with babel's output, and therefore the
`"rootDir"` option in each generated config is set to `"./src"`.
In projects activated for collective typecheck, `.js` may be converted to `.ts`
and/or `.ts` sources may be added without any additional changes needed in
package-level `package.json`.
---
Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into
`packages/core/core` (`embark-core`), refactor other packages' imports
accordingly, and delete `packages/core/typings` from the monorepo. This results
in some similarly named but incompatible types exported from `embark-core`
(e.g. `Events` and `EmbarkEvents`, the latter being the one from
`packages/core/typings`); future refactoring should consolidate those types. To
avoid circular dependency relationships it's also necessary to split out
`Engine` from `embark-core` into its own package (`embark-engine`) and to
introduce a bit of duplication, e.g. the `Maybe` type that's now defined in
both `embark-i18n` and `embark-core`.
In the process of the types reorg, move many dependencies spec'd in various
`package.json` to the `package.json` of the package/s that actually depend on
them, e.g. many are moved from `packages/embark/package.json` to
`packages/core/engine/package.json`. Related to those moves, fix some Node.js
`require`-logic related to bug-prone dependency resolution.
Fix all type errors that appeared as a result of activating collective
typecheck across the whole monorepo.
Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors.
Tidy up and add a few items in the root `package.json` scripts.
Bump lerna from `3.16.4` to `3.19.0`.
Bumpt typescript from `3.6.3` to `3.7.2`.
Bumpt tslint from `5.16.0` to `5.20.1`.
Make various changes related to packages' `import`/`require`ing packages that
weren't spec'd in their respective `package.json`. More refactoring is needed
in this regard, but changes were made as the problems were observed in the
process of authoring this PR.
[config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
|
|
|
"rimraf": "3.0.0",
|
2019-02-12 14:02:54 +00:00
|
|
|
"zeppelin-solidity": "1.12.0"
|
build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
},
|
2019-09-27 17:58:08 +00:00
|
|
|
"license": "MIT",
|
build: make DApp templates member packages of the monorepo
Previously, templates were in a subdirectory of `packages/embark`. Reorganize
them so that they are member packages of the monorepo. This allows them to
cleanly depend on other members of the monorepo,
e.g. `embarkjs-connector-web3`.
It is desirable for the templates, in the context of the monorepo, to specify
embark as a dependency, to take advantage of `npx embark test` (and it's a
"forward looking" setup re: how we plan to evolve embark). However, if embark
were to specify the template packages as dependencies a circular relationship
would be introduced, which is [unsupported by Lerna][circular]. Therefore,
revise the template generator so that all templates are resolved / fetched at
runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer
"built-ins" *per se*. This change won't be apparent to embark's users, but it
does mean that the template generator won't work (in a production install of
embark) if it can't connect to the npm registry, i.e. when the user runs
`embark demo` or `embark new [--simple]`. When embark is inside the monorepo,
templates are resolved and copied from the yarn workspace rather than being
fetched from the registry, which is convenient for development. Also, any
template dependencies that are members of the monorepo are linked into the
copied template's `node_modules` rather than being installed from the registry,
again for convenience. During template generation, remove scripts and
dependencies that pertain only to membership in the monorepo; for now, that
involves removing embark as a dependency since we're not quite ready for that
arrangement to be the default, i.e. outside of the monorepo.
Refactor the root scripts so that more of them can consistently be used with
Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that
don't support filtering generally have a `:full` postfix.
Flip `clean` and `reset` scripts at the root and in the member packages for
consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have
each package run its `reset` script when its `clean` script is invoked (and
that's all for now), relying on `lerna clean` to delete packages'
`node_modules` in view of how Lerna's topological sorting works.
Lift the implementation of `embark reset` into a private package in
`packages/embark-reset` and make it a bundled dependency of embark. Packages in
`dapps/*` depend on `embark-reset` directly and make use of it with `npx
embark-reset` (but only in monorepo context). This removes a "wart" where
reboots could show errors when embark's sources aren't already built in
`packages/embark/dist`. Users will not notice any difference since `embark
reset` works as before, transparently making use of the `embark-reset`
package. The only downside to having it be a bundled dependency of embark is
that bundled deps have all of their `node_modules` included in the tarball
built with `npm pack` (that's why having the templates as bundled dependencies
of embark isn't a viable approach). However, `embark-reset` only has one
dependency, `rimraf`, which is a tiny module, so the cost seems acceptable.
As part of the reorganization, move `test_dapps` into `dapps/tests` and
`packages/embark/templates` into `dapps/templates`. Keep the directory names
short but revise the package names to facilitate simple filtering with
`embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some
redundant ignore listings.
Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid
conflicts that could arise over network ports. The `ci:full` and `qa:full`
scripts use `--concurrency=1` in all scopes, for two reasons: resource
limitations on Travis and AppVeyor result in slower runs with concurrency >1,
and if something fails in those contexts it's easier to see what went wrong
when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`.
Bump the Lerna version.
[circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
|
|
|
"name": "embark-dapp-test-app",
|
build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
"private": true,
|
|
|
|
|
"scripts": {
|
build: merge test_dapps with the root monorepo
[PR 1318][PR1318] introduces a monorepo member that's used as a DApp
dependency, but the present arrangement whereby `test_dapps/` is an embedded
monorepo makes it difficult to develop and test such packages in tandem with
changes to `test_dapps/packages/*`.
Reorganize `test_dapps/*` as members of the root monorepo and make various
adjustments accordingly. This makes it easy to develop test dapps and any
packages they depend on simultaneously, but we lose the CI/QA arrangement where
test dapps are run with an embark installed from fresh tarballs. That
arrangement, which is quite worthwhile as a means to detect problems arising
from transitive dependencies as soon as possible, will be re-introduced in
another PR, possibly involving an auxiliary repository such as
embark-framework/dapp-bin.
Since the `package.json` `"test"` scripts of `test_dapps/*` are now kicked off
as part of `yarn test` in the root, and since port allocation isn't fully
dynamic, it's important to run `yarn test` with `lerna run`'s `--concurrency=1`
option. For the same reasons, the root `ci` and `qa` scripts are similarly
restricted. In the future, this setup can be refactored and improved, along
with use of `lerna run`'s `--since` option to increase testing efficiency in
CI.
[PR1318]: https://github.com/embark-framework/embark/pull/1318
2019-02-12 07:21:04 +00:00
|
|
|
"ci": "npm run qa",
|
build: make DApp templates member packages of the monorepo
Previously, templates were in a subdirectory of `packages/embark`. Reorganize
them so that they are member packages of the monorepo. This allows them to
cleanly depend on other members of the monorepo,
e.g. `embarkjs-connector-web3`.
It is desirable for the templates, in the context of the monorepo, to specify
embark as a dependency, to take advantage of `npx embark test` (and it's a
"forward looking" setup re: how we plan to evolve embark). However, if embark
were to specify the template packages as dependencies a circular relationship
would be introduced, which is [unsupported by Lerna][circular]. Therefore,
revise the template generator so that all templates are resolved / fetched at
runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer
"built-ins" *per se*. This change won't be apparent to embark's users, but it
does mean that the template generator won't work (in a production install of
embark) if it can't connect to the npm registry, i.e. when the user runs
`embark demo` or `embark new [--simple]`. When embark is inside the monorepo,
templates are resolved and copied from the yarn workspace rather than being
fetched from the registry, which is convenient for development. Also, any
template dependencies that are members of the monorepo are linked into the
copied template's `node_modules` rather than being installed from the registry,
again for convenience. During template generation, remove scripts and
dependencies that pertain only to membership in the monorepo; for now, that
involves removing embark as a dependency since we're not quite ready for that
arrangement to be the default, i.e. outside of the monorepo.
Refactor the root scripts so that more of them can consistently be used with
Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that
don't support filtering generally have a `:full` postfix.
Flip `clean` and `reset` scripts at the root and in the member packages for
consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have
each package run its `reset` script when its `clean` script is invoked (and
that's all for now), relying on `lerna clean` to delete packages'
`node_modules` in view of how Lerna's topological sorting works.
Lift the implementation of `embark reset` into a private package in
`packages/embark-reset` and make it a bundled dependency of embark. Packages in
`dapps/*` depend on `embark-reset` directly and make use of it with `npx
embark-reset` (but only in monorepo context). This removes a "wart" where
reboots could show errors when embark's sources aren't already built in
`packages/embark/dist`. Users will not notice any difference since `embark
reset` works as before, transparently making use of the `embark-reset`
package. The only downside to having it be a bundled dependency of embark is
that bundled deps have all of their `node_modules` included in the tarball
built with `npm pack` (that's why having the templates as bundled dependencies
of embark isn't a viable approach). However, `embark-reset` only has one
dependency, `rimraf`, which is a tiny module, so the cost seems acceptable.
As part of the reorganization, move `test_dapps` into `dapps/tests` and
`packages/embark/templates` into `dapps/templates`. Keep the directory names
short but revise the package names to facilitate simple filtering with
`embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some
redundant ignore listings.
Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid
conflicts that could arise over network ports. The `ci:full` and `qa:full`
scripts use `--concurrency=1` in all scopes, for two reasons: resource
limitations on Travis and AppVeyor result in slower runs with concurrency >1,
and if something fails in those contexts it's easier to see what went wrong
when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`.
Bump the Lerna version.
[circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
|
|
|
"clean": "npm run reset",
|
build: merge test_dapps with the root monorepo
[PR 1318][PR1318] introduces a monorepo member that's used as a DApp
dependency, but the present arrangement whereby `test_dapps/` is an embedded
monorepo makes it difficult to develop and test such packages in tandem with
changes to `test_dapps/packages/*`.
Reorganize `test_dapps/*` as members of the root monorepo and make various
adjustments accordingly. This makes it easy to develop test dapps and any
packages they depend on simultaneously, but we lose the CI/QA arrangement where
test dapps are run with an embark installed from fresh tarballs. That
arrangement, which is quite worthwhile as a means to detect problems arising
from transitive dependencies as soon as possible, will be re-introduced in
another PR, possibly involving an auxiliary repository such as
embark-framework/dapp-bin.
Since the `package.json` `"test"` scripts of `test_dapps/*` are now kicked off
as part of `yarn test` in the root, and since port allocation isn't fully
dynamic, it's important to run `yarn test` with `lerna run`'s `--concurrency=1`
option. For the same reasons, the root `ci` and `qa` scripts are similarly
restricted. In the future, this setup can be refactored and improved, along
with use of `lerna run`'s `--since` option to increase testing efficiency in
CI.
[PR1318]: https://github.com/embark-framework/embark/pull/1318
2019-02-12 07:21:04 +00:00
|
|
|
"qa": "npm run test",
|
build: implement collective typecheck
This PR replaces #2057.
Implement a collective typecheck action that can be invoked in the root of the
monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`.
Include the watch-mode typecheck action as part of `yarn start` (a.k.a
`yarn watch`).
To activate collective typecheck for a package in the monorepo, its
`package.json` file should specify:
```
{
"embark-collective": {
"typecheck": true
}
}
```
*-or-*
```
{
"embark-collective": {
"typecheck": {...}
}
}
```
Where `{...}` above is a `tsconfig.json` fragment that will be merged into the
config generated for the package according the same rules that `tsc` applies
when merging [configs][config].
When collective typecheck begins, it generates a `tsconfig.json` for the root
of the monorepo and for each package that is activated for the action. If the
generated JSON is different than what's on disk for the respective root/package
config, or if the config is not present on disk, then it will be
written. Changes to generated `tsconfig.json` files should be committed; such
changes will arise when there are structural changes to the monorepo, e.g. a
package is added, removed, moved and/or the directory containing it is
renamed. Since the configs are only generated at the beginning of collective
typecheck, when structural changes are made in the monorepo `yarn typecheck`
(or `yarn start` or `yarn watch:typecheck`) should be restarted.
Nearly all of the packages in the monorepo (i.e. all those for which it makes
sense) have been activated for collective typecheck. Even those packages that
don't contain `.ts` sources are activated because `tsc` can make better sense
of the code base as a whole owing to the project references included in the
generated `tsconfig.json` files. Also, owing to the fully cross-referenced
`tsconfig.json` files, it's possible for `tsc` to type check the whole code
base without babel (`yarn build` or `yarn watch:build`) having been run
beforehand.
**NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive:
on this author's 2019 MacBook Pro it takes around three minutes, the fans spin
up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has
completed, the next full typecheck will complete in a few seconds or less; and
when running in watch-mode there is likewise a *big* speedup once a full
typecheck has completed, whether that full check happened before it's running
in watch-mode or when watch-mode itself resulted in a full check before
switching automatically to incremental check, as well a corresponding *big*
reduction in resource consumption. A full check will be needed any time
`yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh
clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`.
The combination of options in each generated package-level `tsconfig.json` and
the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript
declaration files) into the `dist/` directory of each package. That
output is intended to live side-by-side with babel's output, and therefore the
`"rootDir"` option in each generated config is set to `"./src"`.
In projects activated for collective typecheck, `.js` may be converted to `.ts`
and/or `.ts` sources may be added without any additional changes needed in
package-level `package.json`.
---
Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into
`packages/core/core` (`embark-core`), refactor other packages' imports
accordingly, and delete `packages/core/typings` from the monorepo. This results
in some similarly named but incompatible types exported from `embark-core`
(e.g. `Events` and `EmbarkEvents`, the latter being the one from
`packages/core/typings`); future refactoring should consolidate those types. To
avoid circular dependency relationships it's also necessary to split out
`Engine` from `embark-core` into its own package (`embark-engine`) and to
introduce a bit of duplication, e.g. the `Maybe` type that's now defined in
both `embark-i18n` and `embark-core`.
In the process of the types reorg, move many dependencies spec'd in various
`package.json` to the `package.json` of the package/s that actually depend on
them, e.g. many are moved from `packages/embark/package.json` to
`packages/core/engine/package.json`. Related to those moves, fix some Node.js
`require`-logic related to bug-prone dependency resolution.
Fix all type errors that appeared as a result of activating collective
typecheck across the whole monorepo.
Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors.
Tidy up and add a few items in the root `package.json` scripts.
Bump lerna from `3.16.4` to `3.19.0`.
Bumpt typescript from `3.6.3` to `3.7.2`.
Bumpt tslint from `5.16.0` to `5.20.1`.
Make various changes related to packages' `import`/`require`ing packages that
weren't spec'd in their respective `package.json`. More refactoring is needed
in this regard, but changes were made as the problems were observed in the
process of authoring this PR.
[config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 17:01:38 +00:00
|
|
|
"reset": "npx embark-reset && npx rimraf embark-*.tgz package",
|
build: make DApp templates member packages of the monorepo
Previously, templates were in a subdirectory of `packages/embark`. Reorganize
them so that they are member packages of the monorepo. This allows them to
cleanly depend on other members of the monorepo,
e.g. `embarkjs-connector-web3`.
It is desirable for the templates, in the context of the monorepo, to specify
embark as a dependency, to take advantage of `npx embark test` (and it's a
"forward looking" setup re: how we plan to evolve embark). However, if embark
were to specify the template packages as dependencies a circular relationship
would be introduced, which is [unsupported by Lerna][circular]. Therefore,
revise the template generator so that all templates are resolved / fetched at
runtime, i.e. `boilerplate`, `demo`, and `simple` are no longer
"built-ins" *per se*. This change won't be apparent to embark's users, but it
does mean that the template generator won't work (in a production install of
embark) if it can't connect to the npm registry, i.e. when the user runs
`embark demo` or `embark new [--simple]`. When embark is inside the monorepo,
templates are resolved and copied from the yarn workspace rather than being
fetched from the registry, which is convenient for development. Also, any
template dependencies that are members of the monorepo are linked into the
copied template's `node_modules` rather than being installed from the registry,
again for convenience. During template generation, remove scripts and
dependencies that pertain only to membership in the monorepo; for now, that
involves removing embark as a dependency since we're not quite ready for that
arrangement to be the default, i.e. outside of the monorepo.
Refactor the root scripts so that more of them can consistently be used with
Lerna's filter options, e.g. `--scope` and `--ignore`. "Combo" scripts that
don't support filtering generally have a `:full` postfix.
Flip `clean` and `reset` scripts at the root and in the member packages for
consistency re: Lerna's notion of `clean` and embark's notion of `reset`. Have
each package run its `reset` script when its `clean` script is invoked (and
that's all for now), relying on `lerna clean` to delete packages'
`node_modules` in view of how Lerna's topological sorting works.
Lift the implementation of `embark reset` into a private package in
`packages/embark-reset` and make it a bundled dependency of embark. Packages in
`dapps/*` depend on `embark-reset` directly and make use of it with `npx
embark-reset` (but only in monorepo context). This removes a "wart" where
reboots could show errors when embark's sources aren't already built in
`packages/embark/dist`. Users will not notice any difference since `embark
reset` works as before, transparently making use of the `embark-reset`
package. The only downside to having it be a bundled dependency of embark is
that bundled deps have all of their `node_modules` included in the tarball
built with `npm pack` (that's why having the templates as bundled dependencies
of embark isn't a viable approach). However, `embark-reset` only has one
dependency, `rimraf`, which is a tiny module, so the cost seems acceptable.
As part of the reorganization, move `test_dapps` into `dapps/tests` and
`packages/embark/templates` into `dapps/templates`. Keep the directory names
short but revise the package names to facilitate simple filtering with
`embark-dapp-*`. Consolidate `.yarnrc` and `.gitignore` and clean up some
redundant ignore listings.
Scripts run with `--scope embark-dapp-*` use `--concurrency=1` to avoid
conflicts that could arise over network ports. The `ci:full` and `qa:full`
scripts use `--concurrency=1` in all scopes, for two reasons: resource
limitations on Travis and AppVeyor result in slower runs with concurrency >1,
and if something fails in those contexts it's easier to see what went wrong
when Lerna's output isn't interleaved from a bunch of scripts in `packages/*`.
Bump the Lerna version.
[circular]: https://github.com/lerna/lerna/issues/1198#issuecomment-442278902
2019-02-25 20:47:54 +00:00
|
|
|
"test": "npx embark test"
|
build: merge test_dapps with the root monorepo
[PR 1318][PR1318] introduces a monorepo member that's used as a DApp
dependency, but the present arrangement whereby `test_dapps/` is an embedded
monorepo makes it difficult to develop and test such packages in tandem with
changes to `test_dapps/packages/*`.
Reorganize `test_dapps/*` as members of the root monorepo and make various
adjustments accordingly. This makes it easy to develop test dapps and any
packages they depend on simultaneously, but we lose the CI/QA arrangement where
test dapps are run with an embark installed from fresh tarballs. That
arrangement, which is quite worthwhile as a means to detect problems arising
from transitive dependencies as soon as possible, will be re-introduced in
another PR, possibly involving an auxiliary repository such as
embark-framework/dapp-bin.
Since the `package.json` `"test"` scripts of `test_dapps/*` are now kicked off
as part of `yarn test` in the root, and since port allocation isn't fully
dynamic, it's important to run `yarn test` with `lerna run`'s `--concurrency=1`
option. For the same reasons, the root `ci` and `qa` scripts are similarly
restricted. In the future, this setup can be refactored and improved, along
with use of `lerna run`'s `--since` option to increase testing efficiency in
CI.
[PR1318]: https://github.com/embark-framework/embark/pull/1318
2019-02-12 07:21:04 +00:00
|
|
|
},
|
2020-01-20 15:57:35 +00:00
|
|
|
"version": "5.1.0-nightly.1"
|
build: implement a monorepo with Lerna
TL;DR
=====
`yarn install` in a fresh clone of the repo.
`yarn reboot` when switching branches.
When pulling in these changes, there may be untracked files at the root in
all/some of:
```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```
They can be safely deleted since those paths are no longer in use at the root.
Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:
`yarn build --scope embark` build only `packages/embark`.
`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.
Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.
It's also possible to `cd` into a particular package and run its scripts
directly:
```
cd packages/embark && yarn watch
```
Hot Topics & Questions
======================
What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?
Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?
Scripts
=======
If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.
`yarn build`
------------
Runs babel, webpack, etc. according to a package's `build` script.
`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.
`yarn ci`
---------
Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.
Also runs the `ci` script of the embedded `test_dapps` monorepo.
`yarn clean`
------------
Runs rimraf, etc. according to a package's `clean` script.
`yarn globalize`
----------------
Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).
`yarn lint`
-----------
Runs eslint, etc. according to a package's `lint` script.
`yarn package`
--------------
Invokes `npm pack` according to a package's `package` script.
`yarn qa`
---------
Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.
There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.
The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.
`yarn reboot`
-------------
Invokes the `reset` script and then does `yarn install`.
The `reboot:full` variant invokes `reset:full` and then does `yarn install`.
`yarn release`
--------------
Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).
```
yarn release [bump] [--options]
```
* `[bump]` see [`publish` positionals][pub-pos] and [`version`
positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
[`lerna.json`][lerna-json].
To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:
```
yarn release prerelease --dist-tag next
```
For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.
Condensend instructions:
```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```
`yarn reset`
------------
Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.
The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.
Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.
`yarn start`
------------
Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.
`yarn test`
-----------
Run mocha, etc. according to a package's `test` script.
The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.
`yarn test_dapps`
-----------------
Runs the `test` script of the embedded `test_dapps` monorepo.
The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.
`yarn typecheck`
----------------
Runs tsc, etc. according to a package's `typecheck` script.
Notes
=====
`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.
[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.
All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.
`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).
Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.
In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.
The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.
Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.
`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.
`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".
`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.
`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.
[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-01-28 18:15:05 +00:00
|
|
|
}
|
