174 lines
5.3 KiB
TypeScript
Raw Normal View History

import { each } from "async";
build: implement collective typecheck This PR replaces #2057. Implement a collective typecheck action that can be invoked in the root of the monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`. Include the watch-mode typecheck action as part of `yarn start` (a.k.a `yarn watch`). To activate collective typecheck for a package in the monorepo, its `package.json` file should specify: ``` { "embark-collective": { "typecheck": true } } ``` *-or-* ``` { "embark-collective": { "typecheck": {...} } } ``` Where `{...}` above is a `tsconfig.json` fragment that will be merged into the config generated for the package according the same rules that `tsc` applies when merging [configs][config]. When collective typecheck begins, it generates a `tsconfig.json` for the root of the monorepo and for each package that is activated for the action. If the generated JSON is different than what's on disk for the respective root/package config, or if the config is not present on disk, then it will be written. Changes to generated `tsconfig.json` files should be committed; such changes will arise when there are structural changes to the monorepo, e.g. a package is added, removed, moved and/or the directory containing it is renamed. Since the configs are only generated at the beginning of collective typecheck, when structural changes are made in the monorepo `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) should be restarted. Nearly all of the packages in the monorepo (i.e. all those for which it makes sense) have been activated for collective typecheck. Even those packages that don't contain `.ts` sources are activated because `tsc` can make better sense of the code base as a whole owing to the project references included in the generated `tsconfig.json` files. Also, owing to the fully cross-referenced `tsconfig.json` files, it's possible for `tsc` to type check the whole code base without babel (`yarn build` or `yarn watch:build`) having been run beforehand. **NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive: on this author's 2019 MacBook Pro it takes around three minutes, the fans spin up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has completed, the next full typecheck will complete in a few seconds or less; and when running in watch-mode there is likewise a *big* speedup once a full typecheck has completed, whether that full check happened before it's running in watch-mode or when watch-mode itself resulted in a full check before switching automatically to incremental check, as well a corresponding *big* reduction in resource consumption. A full check will be needed any time `yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`. The combination of options in each generated package-level `tsconfig.json` and the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript declaration files) into the `dist/` directory of each package. That output is intended to live side-by-side with babel's output, and therefore the `"rootDir"` option in each generated config is set to `"./src"`. In projects activated for collective typecheck, `.js` may be converted to `.ts` and/or `.ts` sources may be added without any additional changes needed in package-level `package.json`. --- Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into `packages/core/core` (`embark-core`), refactor other packages' imports accordingly, and delete `packages/core/typings` from the monorepo. This results in some similarly named but incompatible types exported from `embark-core` (e.g. `Events` and `EmbarkEvents`, the latter being the one from `packages/core/typings`); future refactoring should consolidate those types. To avoid circular dependency relationships it's also necessary to split out `Engine` from `embark-core` into its own package (`embark-engine`) and to introduce a bit of duplication, e.g. the `Maybe` type that's now defined in both `embark-i18n` and `embark-core`. In the process of the types reorg, move many dependencies spec'd in various `package.json` to the `package.json` of the package/s that actually depend on them, e.g. many are moved from `packages/embark/package.json` to `packages/core/engine/package.json`. Related to those moves, fix some Node.js `require`-logic related to bug-prone dependency resolution. Fix all type errors that appeared as a result of activating collective typecheck across the whole monorepo. Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors. Tidy up and add a few items in the root `package.json` scripts. Bump lerna from `3.16.4` to `3.19.0`. Bumpt typescript from `3.6.3` to `3.7.2`. Bumpt tslint from `5.16.0` to `5.20.1`. Make various changes related to packages' `import`/`require`ing packages that weren't spec'd in their respective `package.json`. More refactoring is needed in this regard, but changes were made as the problems were observed in the process of authoring this PR. [config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-11 11:01:38 -06:00
import { Callback } from "embark-core";
import { compact, dappPath, isEs6Module, recursiveMerge } from "embark-utils";
import { Logger } from 'embark-logger';
import * as path from "path";
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
import { NodeVM, NodeVMOptions } from "vm2";
const WEB3_INVALID_RESPONSE_ERROR: string = "Invalid JSON RPC response";
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
interface Command {
varName: string;
code: any;
}
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
/**
* Wraps an instance of NodeVM from VM2 (https://github.com/patriksimek/vm2) and allows
* code evaluations in the fully sandboxed NodeVM context.
*/
class VM {
/**
* The local instance of NodeVM that is wrapped
*/
private vm!: NodeVM;
/**
* These external requires are the whitelisted requires allowed during evaluation
* of code in the VM. Any other require attempts will error with "The module '<module-name>'
* is not whitelisted in VM."
* Currently, all of the allowed external requires appear in the EmbarkJS scripts. If
* the requires change in any of the EmbarkJS scripts, they will need to be updated here.
*/
// private _options: NodeVMOptions = {
public _options: NodeVMOptions = {
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
require: {
builtin: ["path", "util"],
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
external: [
"@babel/runtime-corejs3/core-js/json/stringify",
"@babel/runtime-corejs3/core-js/object/assign",
"@babel/runtime-corejs3/core-js/promise",
"@babel/runtime-corejs3/helpers/interopRequireDefault",
// "embark-utils",
// TODO: ideally this shouldnt' be needed/here or should be configurable by the modules themselves somehow
// "embarkjs-ens",
// "embarkjs-ipfs",
// "embarkjs-swarm",
// "embarkjs-whisper",
// "eth-ens-namehash",
// "ipfs-http-client",
fix(@embark/console): Fix console not working with VM2/monorepo The console was not working correctly with the latest VM2/monorepo updates. This PR addresses namely fixes this problem, but also adds a few more notable changes: * SIGNIFICANT improvement in loading time for `embark console` with an already running `embark run`. This is due to removing unneeded services starting, and instead forwarding user input to the main `embark run` process. * All user input commands are now forwarded to the `embark run` process via IPC insteaad of evaluating the command in the `embark console` process. * Removed IPC console history as it's no longer needed due to the above. Side effects: ** The signature of the `runcode:eval` and `runcode:register` events was changed to remove the `toRecord` parameter. ** Old `runcode:eval` signature: `events.request("runcode:eval", "code to be evaluated", (err, result) => {}, isNotUserInput, tolerateError)` ** New `runcode:eval` signature: `events.request("runcode:eval", "code to be evaluated", (err, result) => {}, tolerateError)` ** Old `runcode:register` signature: `events.request("runcode:register", "varName", variableValue, toRecord, (err, result) => {})` ** New `runcode:register` signature: `events.request("runcode:register", "varName", variableValue, (err, result) => {})` * Removed unneeded `forceRegister` flag. * Removed the `VM.getWeb3Config` method as it's no longer being used (EmbarkJS contracts are pulled out from the VM instead). * Updated `web3Connector` `blockchain:connector:ready` to allow for event requests or event emissions. * In the tests, removed the initial `initWeb3Provider` in the `init` as it was being called twice. * In the tests, removed the `web3Connector` check message as the tests are now using the Console, and the console does this check. This was causing duplicate messages to be displayed in the output. * Fix `web3 is not defined` browser error
2019-02-11 15:29:37 +11:00
"rxjs",
"rxjs/operators",
// "web3",
// "swarm-api",
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
],
},
sandbox: { __dirname: dappPath() },
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
};
/**
* @constructor
* @param {NodeVMOptions} options Options to instantiate the NodeVM with.
* @param {Logger} logger Logger.
*/
constructor(options: NodeVMOptions, private logger: Logger) {
this._options = recursiveMerge(this._options, options);
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
this.setupNodeVm(() => { });
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
}
public get options(): NodeVMOptions {
return this._options;
}
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
/**
* Transforms a snippet of code such that the last expression is returned,
* so long as it is not an assignment.
* @param {String} code Code to run.
* @returns Formatted code.
*/
private static formatCode(code: string) {
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
const instructions = compact(code.split(";"));
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
const last = instructions.pop().trim();
const awaiting = code.indexOf("await") > -1;
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
if (!(last.includes("return") || last.includes("="))) {
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
instructions.push(`return ${last}`);
} else {
instructions.push(last);
}
code = instructions.join(";");
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
return `module.exports = (${awaiting ? "async" : ""} function () {${code};})()`;
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
}
/**
* Evaluate a snippet of code in the VM.
* @param {String} code Code to evaluate.
* @param {Boolean} tolerateError If true, errors are logged to the logger (appears in the console).
* @param {Callback<any>} cb Callback function that is called on error or completion of evaluation.
*/
public async doEval(code: string, tolerateError = false, cb: Callback<any>) {
code = VM.formatCode(code);
let result: any;
try {
result = this.vm.run(code, __filename);
} catch (e) {
if (!tolerateError) {
this.logger.error(e.message);
}
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
return cb(e);
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
}
try {
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
result = await result;
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
} catch (error) {
// Improve error message when there's no connection to node
if (error.message && error.message.indexOf(WEB3_INVALID_RESPONSE_ERROR) !== -1) {
error.message += ". Are you connected to an Ethereum node?";
}
fix(@embark/console): Fix console not working with VM2/monorepo The console was not working correctly with the latest VM2/monorepo updates. This PR addresses namely fixes this problem, but also adds a few more notable changes: * SIGNIFICANT improvement in loading time for `embark console` with an already running `embark run`. This is due to removing unneeded services starting, and instead forwarding user input to the main `embark run` process. * All user input commands are now forwarded to the `embark run` process via IPC insteaad of evaluating the command in the `embark console` process. * Removed IPC console history as it's no longer needed due to the above. Side effects: ** The signature of the `runcode:eval` and `runcode:register` events was changed to remove the `toRecord` parameter. ** Old `runcode:eval` signature: `events.request("runcode:eval", "code to be evaluated", (err, result) => {}, isNotUserInput, tolerateError)` ** New `runcode:eval` signature: `events.request("runcode:eval", "code to be evaluated", (err, result) => {}, tolerateError)` ** Old `runcode:register` signature: `events.request("runcode:register", "varName", variableValue, toRecord, (err, result) => {})` ** New `runcode:register` signature: `events.request("runcode:register", "varName", variableValue, (err, result) => {})` * Removed unneeded `forceRegister` flag. * Removed the `VM.getWeb3Config` method as it's no longer being used (EmbarkJS contracts are pulled out from the VM instead). * Updated `web3Connector` `blockchain:connector:ready` to allow for event requests or event emissions. * In the tests, removed the initial `initWeb3Provider` in the `init` as it was being called twice. * In the tests, removed the `web3Connector` check message as the tests are now using the Console, and the console does this check. This was causing duplicate messages to be displayed in the output. * Fix `web3 is not defined` browser error
2019-02-11 15:29:37 +11:00
if (typeof error === "string") {
error = new Error(error);
}
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
return cb(error);
}
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
return cb(null, result);
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
}
/**
* Registers a variable in the global context of the VM (called the "sandbox" in VM2 terms).
* @param {String} varName Name of the variable to register.
* @param {any} code Value of the variable to register.
*/
public registerVar(varName: string, code: any, cb: Callback<null>) {
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
// Disallow `eval` and `require`, just in case.
if (code === eval || code === require) { return; }
// handle ES6 modules
if (isEs6Module(code) && code.default) {
code = code.default;
}
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
this.updateState(() => {
this._options.sandbox[varName] = code;
this.setupNodeVm(cb);
});
}
private updateState(cb: Callback<null>) {
if (!this.vm) { return cb(); }
// update sandbox state from VM
each(Object.keys(this._options.sandbox), (sandboxVar: string, next: Callback<null>) => {
@fix(embark/core): Fix VM2 + Remap Imports + Monorepo + Tests This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together. The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated: Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below). The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere. As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`. --- 1. https://github.com/embark-framework/embark/pull/1286 2. https://github.com/embark-framework/embark/pull/1275 Go to bottom for details --- There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs. 1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`. 2. `embark test --node embark` does not seem to be functioning for reasons unknown. 3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content: ``` pragma solidity ^0.4.24; import "remix_tests.sol"; import "../app/contracts/simple_storage.sol"; contract SimpleStorageTest { SimpleStorage simpleStorage; function beforeAll() public { simpleStorage = new SimpleStorage(100); } function initialValueShouldBeCorrect() public { return Assert.equal( 100, simpleStorage.storedData, "stored data is not what I expected" ); } } ``` After compilation, we would get the error: ``` remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert) return Assert.equal( ^—————^ ``` --- This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing. Related PRs: https://github.com/embark-framework/embark-solc/pull/24 --- Changes include: 1. Add unit tests for recursively remapping imports 2. Handle plugin contracts correctly 3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation. 4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped) 5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned. 6. Handle additional cases for `custom` and http file types. This PR was tested with: - `embark` unit tests - `embark` test_app - `embark` test_app with `embark-solc` plugin - `embark` test_app with `embark-flattener` plugin - `Giveth/lpp-campaign` Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24 When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues: 1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure. 2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`). Fixes for those issues in this PR: 1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider` 2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
2019-02-05 15:38:33 +11:00
this.doEval(sandboxVar, false, (err?: Error | null, result?: any) => {
if (!err) {
this._options.sandbox[sandboxVar] = result;
}
next(err);
});
}, cb);
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
}
/**
* Reinstantiates the NodeVM based on the @type {VMOptions} which are passed in when this
* @type {VM} class is instantiated. The "sandbox" member of the options can be modified
* by calling @function {registerVar}.
*/
private setupNodeVm(cb: Callback<null>) {
this.vm = new NodeVM(this._options);
cb();
feat(@embark/core): Run all code in VM2 All code to be run in the console is run through a completely sandboxed VM2 instance, instead of the default Node VM. VM2 will only allow whitelisted packages in a `require` statement. The whitelisted packages needed to run EmbarkJS scripts are: ``` [ "@babel/runtime-corejs2/helpers/interopRequireDefault", "@babel/runtime-corejs2/core-js/json/stringify", "@babel/runtime-corejs2/core-js/promise", "@babel/runtime-corejs2/core-js/object/assign", "eth-ens-namehash" ] ``` This can be circumvented in an Embark context (ie Plugin) if needed, for example in a Plugin constructor: ``` Embark.events.emit('runcode:register', 'require', require('lodash'), false); Embark.events.request("runcode:eval", "_.head(['a', 'b', 'c', 'd']);", (err, result) => { if(err) return console.log('========> error: ' + err); console.log('========> ' + result); }); ``` Will emit `========> a`. NOTE: Attempts to use this method to override `require` and `eval` should be handled by Embark and not allowed. NOTE: VM2 seems to allow `eval`, however it is in a completely sandboxed environment, so I'm unsure that we need to be too concerned with this. Thoughts? Refactor tests to use standalone instance of the newly created VM class, so that code is not evaluated through the console. This was done based on the new unit test case where accounts are redefined in a subsequent unit test, which was not originally working with the initial VM2 PR. Refactor `codeRunner`, put all code-affecting logic in the `VM` class. Changed `runCode` to `VM` and converted to TypeScript Add unit tests for `VM`.
2018-12-20 22:33:47 +11:00
}
}
export default VM;