Commit Graph

5264 Commits

Author SHA1 Message Date
Michael Bradley, Jr 3f61e314d9 build: implement a common babel config
Setup a `babel.config.js` in the root of the monorepo to be used by
`packages/*`. It won't be used by some packages, e.g. `packages/embark-ui`, but
most of them should use it instead of rolling their own.

Allow for package-level modifications by specifying `babelrcRoots` in the root
`babel.config.js`

Use the babel `--root-mode upward` option in `packages/embark`'s `build`
script. Other packages intending to use the common config should do likewise.

Use a `.babelrc.js` in `packages/embark` to supply the package-specific
`ignore` settings.

Make packages used by the common config devDeps of the root.

Extract babel-related devDeps from `packages/embark`, but don't extract the
non-dev deps since those are used by embark's pipeline in a production
install. Normally, it should only be necessary to have `@babel/cli` and
`@babel/core` in devDeps, and possibly `@babel/runtime-corejs2` in deps, plus
any package-specific babel-related dev/deps. Once we deprecate the pipeline, we
can finish the extraction.

Use `ncu -f '/babel/' -u` to bump the versions of all babel-related deps in the
root and in `packages/embark`. We get better space/time savings from the yarn
workspace when versions match.
2019-02-06 11:34:47 -05:00
Iuri Matias fbc8ab0fea refactor(@embark/embark-typings) move typings to its own module
fix typings

WIP: refactor

WIP: just needed an annotation 🎉 thanks @emizzle!

WIP: don't pack a tarball since it's a private package

WIP: no need to list as a devDep
2019-02-06 11:07:44 -05:00
Michael Bradley, Jr 8091c4e6d0 build: cleanup .gitignore
Put common patterns we expect to use across most `packages/*` into the
top-level `.gitignore`. If there are package-specific patterns they can go in
that package's `.gitignore`. For example, `packages/embark-ui` ignores its
`build/` directory while commonly we ignore the packages' `dist/` directories.
2019-02-05 18:08:14 -06:00
Michael Bradley, Jr 2bf1b15ac3 build: cleanup .yarnrc
Yarn merges `.yarnrc` from the current working directory with `.yarnrc` files
higher in the directory tree, so that file isn't needed in each package.

Unfortunately, npm doesn't do the same for `.npmrc` files, so a similar cleanup
isn't possible.

Add version info to `test_app/extensions/embark-service/package.json` to that
`yarn install` can work correctly if manually invoked in `test_app`.
2019-02-05 14:15:39 -06:00
emizzle 0ba16278bd fix(@embark/test_app): Fix tests as paths have chagned
The latest lerna changes have changed the HTTP paths of some test files used in the repo. This PR updates the paths to their new location.
2019-02-04 22:52:08 -06:00
Michael Bradley, Jr 7a93e4b6a7 build: implement a monorepo with Lerna
TL;DR
=====

`yarn install` in a fresh clone of the repo.

`yarn reboot` when switching branches.

When pulling in these changes, there may be untracked files at the root in
all/some of:

```
.embark/
.nyc_output/
coverage/
dist/
embark-ui/
test_apps/
```

They can be safely deleted since those paths are no longer in use at the root.

Many of the scripts in the top-level `package.json` support Lerna's [filter
options]. For example:

`yarn build --scope embark` build only `packages/embark`.

`yarn build --ignore embark-ui` build everything except `packages/embark-ui`.

Scoping scripts will be more useful when there are more packages in the
monorepo and, for example, `yarn start` doesn't need to be invoked for all of
them while working on just a few of them simultaneously, e.g `embark` and
`embarkjs`.

It's also possible to `cd` into a particular package and run its scripts
directly:

```
cd packages/embark && yarn watch
```

Hot Topics & Questions
======================

What should be done about the [README][embark-readme] for `packages/embark`?
Should the top-level README be duplicated in that package?

Lerna is setup to use [Fixed/Locked mode][fixed-locked], and accordingly
`packages/embark-ui` is set to `4.0.0-beta.0`. The same will be true when
adding embarkjs, swarm-api, etc. to the monorepo. Is this acceptable or do we
want to use [Independent mode][independent]?

Scripts
=======

If a package doesn't have a matching script, `lerna run` skips it
automatically. For example, `packages/embark-ui` doesn't have a `typecheck`
script.

`yarn build`
------------

Runs babel, webpack, etc. according to a package's `build` script.

`yarn build:no-ui` is a shortcut for `yarn build --ignore embark-ui`.

`yarn ci`
---------

Runs a series of scripts relevant in a CI context according to a package's `ci`
script. For `packages/embark` that's `lint typecheck build test package`.

Also runs the `ci` script of the embedded `test_dapps` monorepo.

`yarn clean`
------------

Runs rimraf, etc. according to a package's `clean` script.

`yarn globalize`
----------------

Makes the development embark available on the global PATH, either via
symlink (Linux, macOS) or a shim script (Windows).

`yarn lint`
-----------

Runs eslint, etc. according to a package's `lint` script.

`yarn package`
--------------

Invokes `npm pack` according to a package's `package` script.

`yarn qa`
---------

Very similar to `ci`, runs a series of scripts according to a package's `qa`
script. The big difference between `ci` and `qa` is that at the top-level `qa`
first kicks off `reboot:full`.

There is a `preqa` script ([invoked automatically][npm-scripts]), which is a
bit of a wart. It makes sure that `embark reset` can be run successfully in
`packages/embark/templates/*` when the `reboot` script invokes the `reset`
script.

The `qa` script is invoked by `yarn release` before the latter proceeds to
invoke `lerna publish`.

`yarn reboot`
-------------

Invokes the `reset` script and then does `yarn install`.

The `reboot:full` variant invokes `reset:full` and then does `yarn install`.

`yarn release`
--------------

Works in concert with [lerna publish], which will prompt to verify the version
before proceeding. Use `n` to cancel instead of `ctrl-c` as `lerna publish` has
been seen to occasionally misbehave when not exited cleanly (e.g. creating a
tag when it shouldn't have).

```
yarn release [bump] [--options]
```

* `[bump]` see [`publish` positionals][pub-pos] and [`version`
  positionals][ver-pos]; an exact version can also be specified.
* `--preid` prerelease identifier, e.g. `beta`; when doing a prerelease bump
  will default to whatever identifier is currently in use.
* `--dist-tag` registry distribution tag, defaults to `latest`.
* `--message` commit message format, defaults to `chore(release): %v`.
* `--sign` indicates that the git commit and tag should be signed; not signed
  by default.
* `--release-branch` default is `master`; must match the current branch.
* `--git-remote` default is `origin`.
* `--registry` default is `https://registry.npmjs.org/` per the top-level
  [`lerna.json`][lerna-json].

To release `4.0.0-beta.1` as `embark@next` (assuming version is currently at
`4.0.0-beta.0`) could do:

```
yarn release prerelease --dist-tag next
```

For *test releases* (there is no longer a `--dry-run` option) [verdaccio] and a
filesystem git remote can be used.

Condensend instructions:

```
mkdir -p ~/temp/clones && cd ~/temp/clones
git clone git@github.com:embark-framework/embark.git
cd ~/repos/embark
git remote add FAKEembark ~/temp/clones/embark
```
in another terminal:
```
npm i -g verdaccio && verdaccio
```
in the first terminal:
```
yarn release --git-remote FAKEembark --registry http://localhost:4873/
```

`yarn reset`
------------

Invokes cleaning and resetting steps according to a package's `reset`
script. The big difference between `clean` and `reset` is that `reset` is
intended to delete a package's `node_modules`.

The `reset:full` variant deletes the monorepo's top-level `node_modules` at the
end. That shouldn't be necessary too often, e.g. in day-to-day work when
switching branches, which is why there is `reboot` / `reset` vs. `reboot:full`
/ `reset:full`.

Errors may be seen related to invocation of `embark reset` if embark is not
built, but `reset` will still complete successfully.

`yarn start`
------------

Runs babel, webpack, tsc, etc. (in parallel, in watch mode) according to a
package's `start` script.

`yarn test`
-----------

Run mocha, etc. according to a package's `test` script.

The `test:full` variant runs a series of scripts: `lint typecheck test
test_dapps`.

`yarn test_dapps`
-----------------

Runs the `test` script of the embedded `test_dapps` monorepo.

The `test_dapps:ci` and `test_dapps:qa` variants run the `ci` and `qa` scripts
of the embedded `test_dapps` monorepo, respectively.

`yarn typecheck`
----------------

Runs tsc, etc. according to a package's `typecheck` script.

Notes
=====

`npx` is used in some of the top-level and package scripts to ensure the
scripts can run even if `node_modules` is missing.

[`"nohoist"`][nohoist] specifies a couple of embark packages because
[`restrictPath`][restrictpath] is interfering with access to modules that are
located in a higher-up `node_modules`.

All dependencies in `packages/embark-ui` have been made `devDependencies` since
its production build is self-contained.

`packages/embark`'s existing CHANGELOG's formatting has been slightly adjusted
to match the formatting that Lerna will use going forward (entries in the log
haven't been modified).

Lerna will generate a CHANGELOG at the top-level and in each package. Since
we're transitioning to a monorepo, things may look a little wonky with respect
to old entries in `packages/embark/CHANGELOG.md` and going forward we need to
consider how scoping our commits corresponds to member-packages of the
monorepo.

In `packages/embark`, `test` invokes `scripts/test`, which starts a child
process wherein `process.env.DAPP_PATH` is a temporary path that has all of
`packages/embark/dist/test` copied into it, so that paths to test
helpers/fixtures don't need to be prefixed with `dist/test/` and so that a
`.embark` directory doesn't get written into `packages/embark`.

The `"engines"` specified in top-level and packages' `package.json` reflect a
node and npm pair that match (a source of confusion in the past). The pair was
chosen according to the first post v5 npm that's bundled with node. A
`"runtime"` key/object has been introduced in `packages/embark/package.json`
which is used as the basis for specifying the minimum version of node that can
be used to run embark, and that's what is checked by `bin/embark`.

Some changes have been introduced, e.g. in `lib/core/config` and
`lib/utils/solidity/remapImports` so that it's *not* implicitly assumed that
`process.env.DAPP_PATH` / `fs.dappPath()` are the same as
`process.cwd()`. There are probably several++ places where that assumption is
still in effect, and we should work to identify and correct them.

`embark reset` now deletes `embarkArtifacts/` within a dapp root, and
`embarkArtifacts/` is git-ignored.

`lib/core/env` adds all `node_modules` relative to `process.env.EMBARK_PATH` to
`NODE_PATH` so that embark's modules can be resolved as expected whether
embark's `node_modules` have been deduped or are installed in npm's flat
"global style".

`checkDependencies` has been inlined (see `lib/utils/checkDependencies`) and
slightly modified to support dependencies that have been hoisted into a
higher-up `node_modules`, e.g. as part of a yarn workspace. eslint has been
disabled for that script to avoid more involved changes to it.

`test_apps` is not in `packages/embark`; rather, there is `test_dapps` at the
top-level of the monorepo. `test_dapps` is an embedded monorepo, and its `ci` /
`qa` scripts `npm install` embark from freshly built tarballs of the packages
in the outer monorepo and then use that installation to run `embark test` in
the dapps. This should allow us to rapidly detect breakage related to
auto-bumps in transitive dependencies.

[filter options]: https://github.com/lerna/lerna/tree/master/core/filter-options
[embark-readme]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/README.md
[fixed-locked]: https://github.com/lerna/lerna#fixedlocked-mode-default
[independent]: https://github.com/lerna/lerna#independent-mode
[npm-scripts]: https://docs.npmjs.com/misc/scripts
[lerna publish]: https://github.com/lerna/lerna/tree/master/commands/publish
[pub-pos]: https://github.com/lerna/lerna/tree/master/commands/publish#positionals
[ver-pos]: https://github.com/lerna/lerna/tree/master/commands/version#positionals
[lerna-json]: https://github.com/embark-framework/embark/blob/build/lerna/lerna.json#L11
[verdaccio]: https://www.npmjs.com/package/verdaccio
[nohoist]: https://github.com/embark-framework/embark/blob/build/lerna/package.json#L52-L55
[restrictpath]: https://github.com/embark-framework/embark/blob/build/lerna/packages/embark/src/lib/core/fs.js#L9
2019-02-04 14:28:49 -06:00
Pascal Precht 3eeeec31e4 fix(@embark/solidity): fix binding in method call
This commit fixes a bug where it throws while trying to compile solidity
files as it dereferences its `this`.

Unfortunately passing methods as lambda callbacks doesn't correctly
resolve its `this` scope even within fat arrow functions, resulting in
unexpected behaviour where `this` inside lambda is `undefined`.
2019-02-04 15:18:20 +01:00
emizzle e0fd641df9 fix(@embark/core): fix(@embark/core): Fix recursive import remapping
Remapping of imports was failing if the file had already had it’s import replaced with a pattern that would match with subsequent replacement attempts. For example, if the dapp contract contained
```
import ".embark/node_modules/zeppelin-solidity/contracts/ownership/Ownable.sol”;
```
which lives in `node_modules`, then `zeppelin-solidity/contracts/ownership/Ownable.sol` would be replaced with `.embark/node_modules/zeppelin-solidity/contracts/ownership/Ownable.sol`, resulting in:
```
import ".embark/node_modules/zeppelin-solidity/contracts/ownership/Ownable.sol";
```
On subsequent replacements of the same file, the same replacement would occur, resulting in the incorrect
```
import ".embark/node_modules/.embark/node_modules/zeppelin-solidity/contracts/ownership/Ownable.sol";
```
2019-02-01 11:41:37 +01:00
Anthony Laibe 0ddebc7a80 fix: using async with async npm 2019-01-30 09:53:45 +00:00
Jonathan Rainville 4023392ea9 feat(generattion): remove web3 generation to let EmbarkJS handle it 2019-01-28 16:21:45 -05:00
Jonathan Rainville 52aebebf9e feat(dapp-config): add dapp connection to dapp config (artifact) 2019-01-28 16:21:45 -05:00
Jonathan Rainville 1543f6ad41 feat(contract): contracts can point to an artifact file to bypass deploy and compile 2019-01-25 17:34:21 -05:00
Anthony Laibe daf6e6426b fix: allow to use embark in dev deps 2019-01-25 17:32:23 -05:00
Andre Medeiros 387d33a076 fix: warn when contract bytecode too large for EVM 2019-01-25 17:31:17 -05:00
Andre Medeiros afe2512eff fix: swarm command registers and warns when swarm is disabled 2019-01-25 17:27:56 -05:00
Andre Medeiros e36ea5d24d fix: command handlers named consistently
Now they're named following a `noun:[noun:]verb` convention.
2019-01-25 17:27:39 -05:00
Andre Medeiros 43fed4f3e5 fix(@embark/cockpit): fix gas estimation 2019-01-25 17:27:31 -05:00
Jonathan Rainville 1d459e45b1 feat(deployer): add event to return the contract send object 2019-01-25 17:27:14 -05:00
Iuri Matias d1ff3ed33a feature(@debugger) improve display of debugger variables 2019-01-25 17:24:15 -05:00
Anthony Laibe 0e01e3d846 PR feedback 2019-01-25 16:37:09 -05:00
Anthony Laibe e8da329d60 feat: avoid infinite loop 2019-01-25 16:37:09 -05:00
Anthony Laibe 29db66be23 feat: import resolver 2019-01-25 16:37:09 -05:00
Jonathan Rainville 0d8f23303a fix(generator): fix warnAboutMetamask being undefined 2019-01-25 14:32:05 -05:00
Thiago F. Alencar a3da5c0b5b docs(@embark/): Fixes 'Optionnal' typo in templates/ and contract.js examples 2019-01-25 11:31:03 -05:00
Anthony Laibe 7ac27a0f93 fix: first parallel deploy
Zero is falsy
2019-01-24 12:04:41 -05:00
Anthony Laibe 5968eef349 fix: simulator proxy with ws 2019-01-24 09:08:55 +00:00
emizzle 2613e6d683 feat(@embark/core): Recursively import contracts
Add support to recursively import contracts. If we have three contracts

1. A imports B
2. B imports C

Then prior to this PR, contract A would import contract B, and a remapping would be added to the contract so the compiler would know how to find contract B. However, contract B imports contracts C, and because the `parseFileForImport` method was not recursive, the remappings were not able to go one level deeper to remap the path to contract C, and thus the compiler would not know how to locate contract C, and would complain with the error `File outside of allowed directories.`

With the introduction of this PR, the `parseFileForImport` method is now recursive, and so any contract imported is also checked for it's own imports that can be remapped. Specifically, this use case is applicable when there is a dependency containing contracts that imports one of it's own dependency's contracts, ie:

```
pragma solididty ^0.5.0;

import "dependency-1/contract-1.sol";
```
where the dependencies look like:
```
|- node_modules
|--- dependency-1
|----- contract-1.sol <--- contains import "dependency-2/contract-2.sol"
|--- dependency-2
|----- contract-2.sol
```

Add unit tests that verify recursive imports work.
Add embark depdendency that installs a contract used in the recursive unit tests.
2019-01-22 14:46:38 +11:00
emizzle 02305fa4e5 fix(@embark/core): Fix contract testing with `remix_tests`
Remix_test-injected contract files were not correctly being tested due to an update in the `remix-tests` signature. I'm really not sure how tests were not completely bombing out during testing.

This PR introduces a few changes:

1. Ensure all contract files have been compiled before skipping compilation. This can occur if contract files have been added after a compilation round has already completed (ie in the tests).
2. Update solc tests to support latest `remix_tests.runTests` signature and include `userdoc` in compiled contracts.
3. Ensure `EmbarkJS` is reset before executing solc tests - like in the js tests, this is required for built EmbarkJS contract objects to behave correctly when it's functions are executed against the chain.
4. Refactor “all files already compiled” function.

BLOCKER: base branch `feat/replace-node-vm` must be merged in https://github.com/embark-framework/embark/pull/1234 before this can be merged.
2019-01-22 14:46:06 +11:00
emizzle c1a5bfee3c feat(@embark/core): Improve VM
Improve support for external requires. Allows external requires to be called like so:
```
Embark.events.on('runcode:ready', () => {
    Embark.events.emit('runcode:register', 'generateClass', require('eth-contract-class'), false);

    Embark.registerCustomContractGenerator(contract => {
      return `
        ${contract.className} = generateClass(${JSON.stringify(contract.abiDefinition)}, '${contract.code}');
        ${contract.className}Instance = new ${contract.className}(web3, '${contract.deployedAddress}');
        `;
    });
  });
  Embark.events.once('contracts:deploy:afterAll', () => {
    Embark.events.request('runcode:eval', 'SimpleStorageInstance', (err, SimpleStorageInstance) => {
      if(err) return console.error(err);
      SimpleStorageInstance.get().then((result) => {
        console.log(`=====> SimpleStorageInstance.get(): ${result}`);
      });
    });
  });
```

* Add `runcode:ready` that is fired when the VM is ready to accept registration of variables.
* Add support for registration of ES6 modules
* Add callback to `registerVar` and `setupNodeVm` in the `VM` class.
* Add support for retaining modified sandbox values across new VM instances.
* Add VM unit tests for external reaquries and modified sandbox state.
2019-01-22 10:26:32 +11:00
Jonathan Rainville df872fdd5b feat(compiler): add a new compiler api that checks for compatibility 2019-01-18 09:03:39 -05:00
Pascal Precht e72d6483df feat(@cockpit/transaction-decoder): allow for decoding raw transaction hashes 2019-01-17 10:40:33 +01:00
Anthony Laibe eab9aa57d7 feat(embark-ui): add storybook 2019-01-16 09:03:57 +00:00
Anthony Laibe 85db678f89 chores: rename env var 2019-01-16 09:03:46 +00:00
emizzle 9a9eb45836 feat(@embark/core): Run all code in VM2
All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM.

VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are:
```
[
  "@babel/runtime-corejs2/helpers/interopRequireDefault",
  "@babel/runtime-corejs2/core-js/json/stringify",
  "@babel/runtime-corejs2/core-js/promise",
  "@babel/runtime-corejs2/core-js/object/assign",
  "eth-ens-namehash"
]
```

This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor:
```
Embark.events.emit('runcode:register', 'require', require('lodash'), false);

Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => {
  if(err) return console.log('========> error: ' + err);
  console.log('========> ' + result);
});
```
Will emit `========> a`.

NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed.

NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts?

Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR.

Refactor `codeRunner`, put all code-affecting logic in the `VM` class.

Changed `runCode` to `VM` and converted to TypeScript

Add unit tests for `VM`.
2019-01-16 10:16:54 +11:00
Jonathan Rainville bdb6719baf Revert "feat(blockchain/config): adds a cors command to add it to the config"
This reverts commit ebafe45f22.
2019-01-14 08:29:21 -05:00
Jonathan Rainville 84a74ac4a2 feat(blockchain/config): adds a cors command to add it to the config 2019-01-14 08:29:21 -05:00
Jonathan Rainville 1bafc41d97 change to use an object instead and change templates 2019-01-14 08:29:21 -05:00
Jonathan Rainville 9e349ff267 feat(blockchainConfig): enable having auto cors plus other origins 2019-01-14 08:29:21 -05:00
Jonathan Rainville 540ff751a9 fix(code-generator): use isDev instead of checking env 2019-01-14 10:33:42 +01:00
Anthony Laibe 1e4eaa5fb6 fix: allow to debug instanceOf contracts 2019-01-14 10:33:08 +01:00
emizzle 518d319917 fix(@embark/ipfs): Update IPFS config CORS with default config
Check if IPFS config has `API.HTTPHeaders.Access-Control-Allow-Origin` before attempting to update it.

`ipfs init` produces a default configuration without a `API.HTTPHeaders.Access-Control-Allow-Origin` element in the JSON. This caused an error to be thrown when attempting to update the IPFS config to provide CORS values.
2019-01-13 15:51:34 +01:00
Eric Mastro 05803fcf42
Merge pull request #1220 from embark-framework/feat/enable-regular-txs
feat(@embark/core): Disable regular txs until needed
2019-01-11 23:38:50 +11:00
Anthony Laibe 52d830a34b feat: allow to stop debugger 2019-01-11 11:46:50 +00:00
Iuri Matias 2d716e1101 chore(release): 4.0.0-beta.0 2019-01-10 13:07:10 -05:00
Anthony Laibe 34f5f97b28 feat: add option --no-single-use-cockpit-token
This option is added to run and console.
This can be usefull while developing on cockpit,
the auto reload don't always ask to authenticate
2019-01-10 12:05:01 +00:00
emizzle 135fde0a85 feat(@embark/core): Disable regular txs until needed
Regular transactions (aka “dev funds”) exist in embark as a workaround to a known bug in geth when using metamask. The workaround is to send a transaction at a regular interval (1.5s), which pushes through any transactions that were stuck. The problem is that the transaction logs and trace logs become cluttered and difficult to parse visually.

This PR disables regular transactions until the following conditions are met:
1. Embark is running geth
2. The user is running metamask in their browser
3. The user authenticates to the cockpit with `enableRegularTxs=1|true` in the query string.

A console warning is show in large letters in the browser with a link to the cockpit URL that includes the special query string to enable regular txs.

This could be extended later to have a button in the cockpit that start/stops regular txs. Or at least extended to allow disabling of regular txs once started.

Support standalone blockchain process.
2019-01-10 12:43:18 +11:00
Anthony Laibe 8efa8895aa feat: allow cockpit with docker 2019-01-09 17:35:28 -05:00
Anthony Laibe 2505fa5284 feat: add development mode to cockpit 2019-01-09 17:34:13 -05:00
emizzle 2613c56da7 feat(@embark-ui): Change page title and description
Update page title and description tags in the page head when the view changes.
2019-01-09 17:27:55 -05:00
Jonathan Rainville d0711305fc fix(contracts): fix linking libraries with long paths using output 2019-01-09 17:13:08 -05:00