In addition to `EmbarkJS.Storage` not being available in the console (ie `await EmbarkJS.Storage.isAvailable() alway returned false), the error `Could not connect to a storage provider using any of the dappConnections in the storage config` would always appear in the console. The storage operations in the dapps were working OK.
The fix to this was three-fold:
1) Wait for the ipfs process to be started before attempting to run the `EmbarkJS.Storage.registerProvider/setProvider` in the console.
2) Wait for `EmbarkJS.Storage.registerProvider` to be called before `EmbarkJS.Storage.setProvider`. This was actually handled in the previous PR.
3) Remove any async operations from the `setProviders` method in the storage module. This was causing `callback is not defined` errors which were being swallowed and masqueraded as an unsuccessful attempt to connect to a `dappConnection` or `upload` config.
`EmbarkJS.Blockchain.connectConsole` has a potential race condition with both `cb` and `doneCb` firing at the end of the method.
This PR is an attempt to fix that by first awaiting the `cb`, then finally calling `doneCb`, as suggested by @michaelsbradleyjr in https://github.com/embark-framework/embark/pull/1319#discussion_r256850820
This PR introduces a large number of changes (for that, I am very sorry). Breaking this up in to smaller PR's was causing tests to fail, and the likelihood of them getting merged diminished. There a couple of PRs this PR closes, and as such, I have kep the original commits to preserve the history. The first two (of three) commits are the close PRs, and the last is the glue bringin it all together.
The main goal of this PR is to fix the fragility of the tests with EmbarkJS, however in doing so, a number of recent features have been updated:
Remapping of imports still had a few edge cases that needed to be ironed out, as well as have unit tests created for them. More details of the changes an be seen in the closed PR (below).
The main issue with VM2 was running the code generated EmbarkJS code inside the VM, and getting EmbarkJS contracts out of the VM and available to the tests. This fixed issues where ENS may not have been available to the tests. Notable additions include adding `EmbarkJS.Blockchain.connectTests` to the tests lifecycle, and ensuring `EmbarkJS` is only every required in the console module and not used or passed around elsewhere.
As mentioned above, the main issue with the tests in the context of a monorepo and with embark running as module inside the dapp’s `node_modules`, there were issues getting the correct contract state available inside of the tests. For some reason, this particular case was causing the tests to fail, with ENS never being available (assuming this to be an issue with `EmbarkJS.Blockchain.connect()` never being called). The main fix for this came with passing `web3` as an option in to `EmbarkJS.Blockchain.setProvider()`.
---
1. https://github.com/embark-framework/embark/pull/1286
2. https://github.com/embark-framework/embark/pull/1275
Go to bottom for details
---
There are few known issues with this PR. Instead of trying to fix all of them with this PR, I was hoping to get these issues tackled in separate PRs.
1. `deployIf` directive on contracts defined in the config are not working, however the tests are passing. The issue is that if `ContractA` has a `deployIf` set to `!!ContractB.options.address`, it means that it will not deploy if `ContractB` is not deployed. However, it appears that `ContractA` is attempted to be deployed prior to `ContractB` being deployed, and therefore `ContractA` fails to deploy. Instead, because `ContractA` depends on `ContractB`, `ContractB` should be deployed before `ContractA`.
2. `embark test --node embark` does not seem to be functioning for reasons unknown.
3. Remix tests: Currently there is support for adding contract tests that get process by `remix-tests`, however, there is an error that I believe is not due to embark, but due to the Assert library. For example, if we add a `test/remix_test.sol` to the `test_app` with the following content:
```
pragma solidity ^0.4.24;
import "remix_tests.sol";
import "../app/contracts/simple_storage.sol";
contract SimpleStorageTest {
SimpleStorage simpleStorage;
function beforeAll() public {
simpleStorage = new SimpleStorage(100);
}
function initialValueShouldBeCorrect() public {
return Assert.equal(
100,
simpleStorage.storedData,
"stored data is not what I expected"
);
}
}
```
After compilation, we would get the error:
```
remix_test.sol:14:12: TypeError: Member "equal" not found or not visible after argument-dependent lookup in type(library Assert)
return Assert.equal(
^—————^
```
---
This branch is based off of ()`refactor/embarkjs-in-monorepo`)[https://github.com/embark-framework/embark/tree/refactor/embarkjs-in-monorepo], which does not have passing tests due to the `EmbarkJS` ENS issue mentioned above. However, you should (hopefully) see the tests passing in this branch, meaning if both branches are merged, the tests should be passing.
Related PRs: https://github.com/embark-framework/embark-solc/pull/24
---
Changes include:
1. Add unit tests for recursively remapping imports
2. Handle plugin contracts correctly
3. Allow `prepareForCompilation` to be called from `File`, allowing for external compilers, like `embark-solc` to call this function before compilation.
4. Add flattened remappings to `File` that gets prepared for compilation (ie has it's imports remapped)
5. Return remapped contract content when file type is http. Previously this was broken, as always freshly downloaded (original) content was returned.
6. Handle additional cases for `custom` and http file types.
This PR was tested with:
- `embark` unit tests
- `embark` test_app
- `embark` test_app with `embark-solc` plugin
- `embark` test_app with `embark-flattener` plugin
- `Giveth/lpp-campaign`
Related change to get `embark-solc` up-to-date` with these changes: https://github.com/embark-framework/embark-solc/pull/24
When embark was running as module inside the dapp’s `node_modules`, the tests were failing due to several issues:
1. `web3` was not being set in the global namespace of vm2. `EmbarkJS.Blockchain.setProvider` was therefore failing because it relies on `global.web3` to be set. I guess somehow this works when the test app was running in a child tree of the executing program. maybe this is a security feature of vm2, but i’m not sure.
2. `embarkjs` provider code being injected to the vm twice. This really was the initial point of failure, as this piece of code is requiring embarkjs, so i’m assuming, but again not sure, that maybe it was getting a second instance of `EmbarkJS` which did not have it’s providers set up correctly (hence the error with `ENS provider not set`).
Fixes for those issues in this PR:
1. To circumvent the web3 issue, we are now setting `global.web3` for tests only (the global web3 accessible in the tests), and `web3` is now explicitly passed in to `EmbarkJS.Blockchain.setProvider`
2. To fix the `embarkjs` code being called twice, we are not re-injecting this code to the VM during test initialisations
Packages' `"clean"` scripts should run regardless of whether `node_modules` is
in place at the root and/or package levels, but the `embarkjs` package was
missing a needed `npx` in front of `rimraf` in its script.
Also, bump its devDeps version of rimraf to be in line with the rest of
`packages/*` and `test_dapps/*`.
If a function receives a callback argument then it should not return a promise
if the caller's callback will be invoked. Both invoking a callback and
returning a promise can lead to at best confusion (in code review and at
runtime) and at worst non-deterministic behavior, such as race
conditions. Also, a caller supplying a callback may not handle a returned
promise, leading to unhandled rejection errors.
Refactor all readily identified functions where a callback argument can be
supplied but the function returns a promise regardless. Make use of
`callbackify` and `promisify` where it made sense to do so during the
refactoring. Some callsites of the revised functions may have been accidentally
overlooked and still need to be updated. Some functions that take callback
arguments may execute them synchronously, at odds with control flow of a
returned promise (if a callback wasn't supplied). Such cases should be
identified and fixed so that asynchronous behavior is fully consistent whether
the caller supplies a callback or receives a promise.
Make sure promises that pass control flow to a callback ignore rejections,
since those should be handled by the callback.
Don't return promise instances unnecessarily from async functions (since they
always return promises) and change some functions that return promises to async
functions (where it's simple to do so).
Whisper was using an ad hoc promise-like `messageEvents` object. However, that
object behaved more like an observable, since promises either resolve or
reject, and only do so one time. `messageEvents` was also intertwined with
callbacks. Replace `messageEvents` with RxJS Observable. `listenTo` now returns
Observable instances and callers can subscribe to them.
`Blockchain.connect` of embarkjs could suffer from a race condition where tasks
associated with `execWhenReady` might be ongoing when `connect`'s returned
promise resolves/rejects (or a caller supplied callback fires). Attempt to
ensure that returned-promise / supplied-callback control flow proceeds only
after `execWhenReady` tasks have finished. The control flow involved
is... rather involved, and it could use some further review and refactoring.
Bump webpack and the hard-source-plugin for webpack.
[util]: https://www.npmjs.com/package/util
Don't import git history of embark-framework/EmbarkJS, simply copy over the
sources. Modify `package.json`, etc. re: being situated in the monorepo.
Make use of the root babel config but extend with
`packages/embarkjs/.babelrc.js`.
Build `test/` scripts into `build-test/` and git-ignore `build-test/`.
Revise `Blockchain.connect()` so that if the caller supplies a callback then a
promise is not returned.
Revise tests to test `Blockchain.connect()` usage with and without a callback.
Jonathan Rainville
emizzle
Michael Bradley, Jr