* chore: test framework
* chore(@embark/testing): introduce Plugin apis and other changes
* refactor(@embark/deployment): use new testing APIs for tests
* feat(@embark/compiler): support :before and :after hooks on event compiler:contracts:compile
* style: keep root package.json sorted cleanly
* style: formatting with prettier where print-width is 200
Per the suggestion of @iurimatias
* WIP: PR review
* WIP: PR review
* WIP: refactor to check if file.path is within dappPath() or os.tmpdir()
* WIP: PR review
* WIP: use native async functions (and related JS lang constructs) instead of the async library
* WIP: async.eachObject is no longer a dependency
* fix: fix test-app, contracts index file and reload on change
* fix(@embark/cmd_controller): fix missing nodes
Was removed by accident
* feat(@embark/demo): add favicon to embark demo
* chore(@embark/cockpit): change favicon to new Embark logo
* fix(@embark/embarkjs-ens): fix ENS config for embarkjs-ens
* remove comments
When an options object is not supplied the tests fail because the constructor
expects the options argument to be an object, i.e. referencing
`options.plugins` should not cause a TypeError
* chore(@embark/): move embarkjs packages to their own folder
* chore(@embark/): rename embark-ui folder to cockpit
* chore(@embark/): rename packages already in logical folders to remove embark- prefix
chore(@embark/): rename packages already in logical folders to remove embark- prefix
update package.json files to use correct eslint config
remove core/* from package.json workspaces
* fix(@embark/cmd_controller): fix embark blockchain command
* feat(@embark/blockchain): ping endpoint before starting the node
also use web3 instead of function to ping endpoint
* dont override console.error completely
* fix embarkjs generation
fix ens setProvider
fix embarkjs objects
fix generated embarkjs provider
generate contracts
fix embarkjs-ens
* address some of the issues in the code review
* feat(@embark/specialconfigs): introduce dynamic addresses
This commit introduces a new Smart Contract configuration addressHandler
that lets users define a function to "lazily" compute the address of the
Smart Contract in question.
This is useful when a third-party takes care of deploying a dependency
Smart Contract, but the address of that Smart Contract only being available
at run-time.
Example:
```
deploy: {
SimpleStorage: {
fromIndex: 0,
args: [100],
},
OtherContract: {
deps: ['SimpleStorage'],
address: async (deps) => {
// use `deps.contracts.SimpleStorage` to determine address
},
abiDefinition: ABI
},
}
```
In the example above, OtherContract will be deployed after SimpleStorage
because it uses the deps property to define Smart Contracts that it depends
on. All dependencies are exposed on the addressHandler function parameter
similar to deployment hooks.
Closes#1690
There was an error that would display on the second+ run of embark, that was causing by trying to read JSON of an empty file.
The solution was a combination of ensuring the file existed with defaults when enabled, and also ensuring we await the saving of the file.
Included is a bit of a refactor of how the tracking functions handled the “current chain”. Hopefully, this should make things more clear.
Tests have been updated accordingly.
* fix embarkjs generation
fix ens setProvider
fix embarkjs objects
fix generated embarkjs provider
generate contracts
fix embarkjs-ens
* address some of the issues in the code review
* address some of the issues in the code review
* address some of the issues in the code review
* address some of the issues in the code review
* refactor(@embark/cmd-controller): re add cargo for file watcher
(cherry picked from commit 5c77b4000742a14f42a6dc5c145672b9cd0957a3)
* use generate:all instea dof the event
This is crucial as the storage module tries to generate/eval artifacts that
depend on a `web3` object inside the VM scope. The registration of that object
can only be garuanteed when the `blockchain:started` event has been fired.
This commit ensures these modules are not spinned up in parallel as we're running
into race conditions.
DApps generated with `new --simple` don't have a blockchain config so ensure
that the blockchain proxy and listener (dev_txs) use appropriate provider
settings when setting up web3.
Without these changes, in a fresh `embark new --simple` project the `embark
blockchain` command will encounter unhandled promise rejection with respect to
the proxy; `embark run` will encounter similar with respect to dev_txs.
Previously, when using "development embark" (i.e. `embark` command in the
monorepo) the template generator would always use `yarn install` to install a
template's dependencies. However, if a template relies on `package-lock.json`,
as is the case for `embark-create-react-dapp-template`, that behvaior can cause
serious problems. So, have the choice of install command depend not only on
whether we're using "development embark" but also whether the template has a
`package-lock.json` file.
NOTE: this change can probably result in problems given that `yarn install` and
`npm install` behave differently re: symlinks in `node_modules`. However, such
problems would never show up with production installs of the `embark`
packaage. Moreover, after dropping `embarkjs-connector-web3` we don't presently
have a situation where a monorepo package would be `yarn link`ed into an
instantiated template's `node_modules`. So, in effect, the changes introduced
in this PR allow existing templates to work correctly whether using production
or monorepo embark. The impending embark v5 refactor should deprecate `embark
new` in favor of the yet unfinished `embark init`.
When passing the name of a build pipeline that doesn't exist, e.g.
`embark run|build --pipeline foo`, `writeStats` would cause an unhandled
promise rejection because its config argument is falsy. Checking for that and
returning early from `writeStats` allows the `"no webpack config has the name
'foo'"` error to be reported correctly.
The ignore paths for the default webpack config and babel-loader-overrides
helper script weren't properly adjusted after the legacy pipeline was split-out
into its own package in the monorepo.
Upgrade all dependencies on web3/web3-* v1.0.0-beta.37 to v1.2.1.
Make various adjustments related to the previous convention of
`"web3": "1.0.0-beta"` in `embark.json` signifying that embark's own web3
dependency should be used in dapp builds.
Fix bugs in library manager, including a switch from using the
live-plugin-manager package to using npm in a child process to install
`"versions"` dependencies specified in `embark.json` when a specified version
doesn't match up with embark's own version for that package.
Avoid race conditions when installing `"versions"` by completing all installs
prior to starting other services. If an install fails, then after all the
installs have completed or failed the embark command will exit with error.
Change various comments and update docs to reflect the new default of web3
v1.2.1.
In https://github.com/embark-framework/embark/commit/87d92b6091 we've introduced a feature in our test runner
to report exact gas costs used when calling Smart Contract methods that
cause computation.
Embark keeps a list of deployed contracts in memory and for all transactions
happening during tests, it'll match them to the contracts that performed them
to produce the report logs.
Unfortunately, we've been [resetting that memory](https://github.com/embark-framework/embark/commit/87d92b6091#diff-92b4f79a0473160fe700440b1ced5204R140) of deployed contracts
after every test, making it practically impossible for Embark to find
matching contracts for any transactions, because contracts aren't necessarily
redeployed per spec, resulting in no additional transaction logs whatsoever.
This commit ensures that the memory is never erased and at the same time
ensures it's not leaking infinitely in case multiple contracts are
deployed multiple times in the same process over several specs.
We've introduced a regression in 536a4029ba where the default
`value` sent to payable Smart Contract methods is `-1`, resulting in errors as it's not
a valid value.
For payable methods inside of the contract interatction section of cockpit, the value input has been updated to pass wei to the API. The input field now accepts value like `100 ether` or 25 szabo`. The value entered is automatically converted to wei and shown to the user in real time.
Additionally, the input is validated for a correct value, with an error shown to the user for incorrectly entered values.
A tooltip has been added to help the user enter correct values.
UI updates can be seen in video here: https://monosnap.com/file/642cHH2HxDeiFLzB2VLqHP9GuqoRfz
This is due to a bug that has been introduced in d10c0b7951 where we end up
sending values as numbers to web3's `toWei()` utility. The `value` is always sent as number
and in fact always defined, so we just need to check for it's type and convert it to a string
according to `toWei()`'s API.
When running tests against non-simulated blockchain nodes, even for simplex
Smart Contracts, deployment transactions would exceed the block gas limit.
E.g. running `embark blockchain` in one process and `embark test --node embark`
in another, inside our demo application, will throw an error when Embark attempts
to deploy its `SimpleStorage`:
```
Compiling contracts
Compilation done
[SimpleStorage]: error deploying =SimpleStorage= due to error: Returned error: exceeds block gas limit
Error deploying contracts. Please fix errors to continue.
Error deploying contracts. Please fix errors to continue.
terminating due to error
Error deploying contracts. Please fix errors to continue.
```
The reason for that is because in https://github.com/embark-framework/embark/pull/1650, we've introduced a static
gas estimation for Smart Contract deployment that is just right below Ganache's
maximum gas limit of `6721975`, since Ganache tends to underestimate gas for
complex Smart Contracts due to its [low base fee](8ad1ab29de/lib/utils/gasEstimation.js (L33-L39)).
The static gas estimation would apply any time we're in a test context, but we
didn't take into account the case where tests are executed against nodes
other than the simulated environment.
As mentioned in the comments in the linked PR:
> If this is not spec'ed at all, I wonder what complications it could cause when
> at some point we maybe switch to not using Ganache anymore for tests, or even
> the user itself (which I think is a reasonable thing to do).
This causes the error described above because we easily reach the block gas limit
with just two Smart Contracts and Embark already deploys a few Smart Contracts for
ENS.
So basically what we want is to use the static gas estimation when we know
the node we're connecting to is Ganache. In all other cases we can rely on the
standardized gas estimation offered by the node.
Adjust the API endpoints to augment transaction objects with a timestamp
property from their corresponding blocks. This removes the necessity to copy
the timestamp property from a block to its transactions in the client.
Introduce a `formatTimestampForDisplay` util function in Cockpit for
consistently transforming timestamps into relative or absolute dates, depending
on whether a date is sometime during the last day.
The property is set/available on EmbarkJS in the artifacts, i.e. in a browser
build; but sometime since v3.2.4 it was no longer availble in the cli dashboard
environment.
`EmbarkJS.Messages.isAvailable()` in some cases return synchronously (when whisper isn't
set up), in other cases asynchronously. This actually breaks our demo application for
the following reason:
We check for Whisper's availability via:
```
EmbarkJS.Messages.Providers.whisper.getWhisperVersion((err, _version) => {
if (err) {
return console.log(err);
}
this.setState({whisperEnabled: true});
});
```
There's a couple of problems here:
- This code will break right away when whisper isn't available, resulting in an error:
```
Cannot read property _requestManager of undefined
```
- The reason this error happens is because there's no `web3` object available inside
our EmbarkJS.Messages code. Even though there **is** a web3 object, EmbarkJS.Messages
doesn't know about this because it only sets it when its `setProvider()` API is called,
which effectively doesn't happen at all when Whisper isn't enabled on the connected
node
- While this could be fixed with a simple check on whether EmbarkJS.Messages' internal
`web3` references is a thing, really what should be used in the demo is the `isAvailable()`
API.
`isAvailable()` should always return a promise (similar to `EmbarkJS.Storage.isAvailable()`.
This commit ensures that `isAvailable()` always returns a promise and changes the demo
template to use `isAvailable()` over `getWhisperVersion()`.
Effectively deprecate the `embarkjs-connector-web3` package but don't introduce
a breaking change by simply not loading the plugin if it's specified in a
DApp's `embark.json`. If the deprecated plugin is specified, display a message
indicating the plugin was ignored and suggesting it be removed from the
project's `embark.json` and `package.json`.
This PR adds a command to get full account details from the contradts config that includes info like private key.
The existing, and similar command, `blockchain:provider:contract:accounts:get` would only return account addresses if they existed, and not the full account info.
This check was already made when sending messages to whisper channel, however, we didn't
perform the same check when subscribing to channels within cockpit.
The webserver's job is to serve files generated by Embark's built-in pipeline,
however, since v4 users can choose they front-end tool to take care of building,
bundling and packing their DApps. Usually these tools come with a built-in
dev server as well.
Therefore, when the pipeline is turned off (which also soon will be the default),
there's not need start a webserver.
In https://github.com/embark-framework/embark/pull/1119 we've introduced a feature where
users can provide an already compiled ABI for Smart Contracts so they can be used
without the need to compiling them again.
This also means that, internally, those Smart Contract object won't have any
bytecode attached to it.
Later on, in 387d33a076, we've introduced a warning
which is rendered when a Smart Contract's bytecode is too large. The check expects
a Smart Contract object to have bytecode associated to it, which will break the code
in cases where a Smart Contract has already an ABI and therefore didn't need compilation.
This commit ensures we only check a Smart Contract's bytecode when bytecode exists for
the Smart Contract in question.
When trying to either sending, or listening to whisper channels within
Embark's consoles (CLI and Cockpit), the console outputs an error that
`EmbarkJS` isn't defined.
E.g. running:
```
> EmbarkJS.Messages.listenTo({ topic: ['somechannel'])
```
Outputs:
```
EmbarkJS is not defined
```
This seemed very odd as outputting `EmbarkJS` and all of its members
worked totally fine.
It turns out that both methods, `listenTo()` and `sendMessage()`, in
`EmbarkJS.Messages` are not necessarily what one thinks they are.
EmbarkJS decorates both methods to create some options that need to be
available in the delegate, two of them being `EmbarkJS.Utils.toAscii`
and `EmbarkJS.Utils.fromAscii` (ac76a40a61/packages/embarkjs-whisper/src/index.js (L43-L62) and ac76a40a61/packages/embarkjs-whisper/src/index.js (L64-L73))
These two global references to `EmbarkJS` are the only ones in `embarkjs-whisper`
and they are not the same reference as the `EmbarkJS` sandbox global
registered in the VM here ac76a40a61/packages/embark-code-runner/src/index.ts (L33).
In other words, the `EmbarkJS is not defined` message actually refers
to the `EmbarkJS` in the wrapping method, not the one registered with
the VM, which is also why inspecting the `EmbarkJS` object through the
VM works fine.
Since the `toAscii()` and `fromAscii()` methods in use are really just
facades around `web3.utils.[fromAscii|toAscii]()`, we can replace the
global EmbarkJS dependency with web3 utils that are already available
anyways.
Upgrade chokidar to a version that's compatible with NodeJS v12.x.
Unfortunately, embark has other transitive dependencies that are not compatible
with v12.x, but upgrading chokidar is still a good step.
When running unit tests inside a project that configures ENS subdomains using Smart Contract directives,
the tests will output an error because of that particular Smart Contract's `deployedAddress` being `undefined`.
This happens only when running tests, not when deploying the Smart Contracts including the custom ENS setup.
It turns out that Embark attempts to compile and deploy the Smart Contracts of the project *twice* - once
before tests are executed, and another time **after** tests are done executing.
Both compilations/deployments are triggered through our custom `config()` function within test context,
which ensures all Smart Contracts are deployed before tests are executed.
This explains a few things:
1. There's no such issue when running `embark run`, in fact the custom ENS subdomains work perfectly fine
2. That's also why the tests are passing fine as well as the first compilation/deployment doesn't have any issues.
The errors only appear *after* the tests have been executed.
Still, this raises a few more questions, mainly
- Why is the Smart Contract's `deployedAddress` property `undefined` when `config()` is executed a second time?
- Why is `config()` executed a second time in the first place?
Let's look into both of these.
Assuming that there's a valid reason that `config()` is called twice, it's remains unclear why that property
of a Smart Contract object is `undefined` in the second run. The reason for that is that Embark determines whether
or not a particular Smart Contract should be deployed. One of the routines ensures that only the Smart Contracts
configured for deployment are actually attempted to be deployed (as opposed to just deploying all Smart Contracts
found in the file system).
It turns out that the second `config()` call is done without any Smart Contract configuration, resulting basically in no deployment
for any Smart Contract of the application. The `address` and `deployedAddress` of a Smart Contract object are however only set
*after* deployment, resulting in them being `undefined`.
**All this makes sense.**
`config()` is simply not designed for being used with an empty `contracts: {}` config. This raises another question:
Why is `config()` called with an empty configuration? This leads us to the second point.
It does seem a bit weird that Embark tries to configure compile *and* deploy a DApp's Smart Contracts again right **after**
the tests have been executed. So why is that?
A quick `git blame` (no blame intended here) shows us that this routine has been introduced in https://github.com/embark-framework/embark/commit/12cbb7bdd.
Notice the second list point of the commit:
> Contracts that had been compiled in the JS tests were being deleted at the end of the JS test run, which caused errors
> with the files not being found. The fix was to reset the contracts config to no longer require the non-test contracts
> to be compiled/deployed.
The above is a little tricky to understand without a little bit of context: Embark runs two types of tests, JavaScript tests
and Solidity tests. It does that as part of a single process, keeping state in memory in-between those two test runs.
With that in mind, the commit mentioned above says the following:
1. Artifacts that Embark generates as part of its compilation process are erased after JS tests are done executing.
This makes a lot of sense as we don't want to leave any side effects undone when tests are finished.
2. The commit ensures that not only the artifacts are removed from disc, but also updates Embark's state in memory
for `contractFiles`. The reason for that is that otherwise, in the second test run for Solidity tests, Embark will
throw errors as it tries to look up the path for the artifacts in memory for all the Smart Contracts that had been
compiled before.
3. Last but not least, there's *another* state that needs a reset and that's the Smart Contract configuration. If Embark
doesn't reset the memory it'll assume in the second run that all of those Smart Contracts left in memory
"have no code associated to it", while in reality, they shouldn't be in memory in the first place.
So it seems that `config()` is called a second time with an empty Smart Contracts configuration just to ensure that memory
is reset and no error messages are shown.
As discussed earlier, this unfortunately also introduced a lot of side effects along the way as Embark tries to reregister
ENS subdomains from Smart Contracts that are marked as undeployed and therefore don't have an address to interpolate.
While there's probably different ways to go about it, the most straight forward fix is to simply not call `config()`
a second time when it's really not needed. If the goal is to just reset the memory, we can take advantage of
Embark's internal `config:contractsConfig:set` event, which is what this commit is doing.
The embark-utils package, since 4.1.0-beta.1, is a transitive DApp dependency;
but it's loaded in the main process (not a child process), so embark-utils' own
deps can't be indirectly supplied from embark itself via NODE_PATH.
When introducing the `embark-api-client` in https://github.com/embark-framework/embark/commit/c1bbdbf34
we didn't properly update Cockpits apiService, resulting in calls to API endpoints
that don't exist.
This commit fixes a bug where calls to Embark's API to subscribe to whisper
channels are in a broken state.
It also updates Cockpit's apiService to no longer pass ignored parameters to embark-api-client.
With the move of Embark's modules into their own packages, the names under
which they are registered in the API service have changed as well.
This caused Cockpit to no longer being able to properly detect whether
ENS is enabled in the current Embark project
This package comes with a `setUpEnv()` function that can be called to
prepare Embark's environment variables for different process contexts.
This allows us to set up Embark's environment within a test runner context
without the requirement of importing `packages/embark/src/lib/core/env.js`,
which happens to set up those necessary environment variables as a side effect.
It's important that a path to `packages/embark`'s root is passed down to `setUpEnv()`
to ensure `EMBARK_PATH` gets the correct value.
E.g. within a package `embark-console`, `setUpEnv()` can be called within its tests
like the following:
```
// packages/embark-console/src/test/console.js
import { joinPath, setUpEnv } from 'embark-utils';
setUpEnv(joinPath(__dirname, '../../../embark'));
```
Here, `__dirname + '../../../embark'` ensures `EMBARK_PATH` points to the root
of `packages/embark`. This might look different in other contexts.
For example calling this function from a test file within `packages/embark`, this would
look like this:
```
// packages/embark/src/test/some_file.js
import { joinPath, setUpEnv } from 'embark-utils';
setUpEnv(joinPath(__dirname, '../../'));
```
As part of 880a3a6946, we've introduced a
regression where the `colors` package isn't imported in the scope anymore where it's
needed. In this case the pipeline package. This resulted in log output messages
such as:
```
undefinedwriting fileundefined
```
Adding the `colors` package as dependency and importing it accordingly
fixes the issue.
Make the adjustment even if the test script is currently disabled. Consistently
use nyc to generate coverage with mocha even if the test script is currently
disabled.
If the `embark` cli is detected to be inside the monorepo, the template
generator yarn-links any template deps that are part of the monorepo into the
dapp. However, `embarkjs-connector-web3` now has a dependency on `embark-core`,
which in turn has a dependency on `embark-i18n`. That causes `npm install` to
fail inside the dapp because npm checks for deps of `embarkjs-connector-web3`
relative to the dapp instead of checking relative to the symlink. `yarn
install` does not suffer from that behavior so use it instead.
Move the debugger module to the `embark-debugger` package.
Handle case where `debug` is entered as a console command, and there is no transaction to be debugged.
add embark-i18n and mocha deps
Add support for `service api on/off` commands.
Deprecate commands `api start/stop` in favor of `service api on/off`.
`service api on` - Enables the API server serving Cockpit. Shows an error if the API server is already starting or started.
`service api off` - Disables the API server serving Cockpit. Shows an error if the API server is already stopping or stopped.
`api start` - This command has been deprecated in favor of `service api on` and will be removed in future versions.
`api stop` - This command has been deprecated in favor of `service api off` and will be removed in future versions.
`api:start` - This event has been deprecated and will be removed in future versions.
`api:stop` - This event has been deprecated and will be removed in future versions.
Remove support for `service whisper on/off` because Whisper cannot be started without first disabling it in the communications config, then restarting Embark.
Add documentation for the `service [process] on/off` command
This command was only added a few commits back so it is not considered a breaking change.
Refactor the `service [process] on/off` commands so that there is only one command. Previously we had a command registered for each process, ie:
```
service blockchain on/off
service ipfs on/off
…
```
Now, we only have one command with many options:
```
service [process] on/off - Starts/stops the process. Options: blockchain, embark, ipfs, api
```
Whisper is deliberately not included in the available options for the aforementioned reasons.
This enables removing unnecessary `core/fs` dependencies which can be
replaced with `fs-extra`.
This commit also fixes a bug introduced in f868d1216d
where methods from `embark.fs` weren't available.
The reason for that is because we have several *Process instances
that are created through child process communication, specifically
process.send() APIs. Using those APIs, we can only send data structures
however, methods attached on any of those will get lost.
This is the case when sending embark.fs through process.send().
We still need fs in those places though, mostly because they are relying
on embarkPath and dappPath().
These places are now importing those functions from `embark-core`. Other
API such as writeFile or mkdirp() can be accessed through fs-extra
or fs modules.
This pull-request upgrades `ethereumjs-wallet`, which has upgraded the underlying dependency on scrypt.js to 0.3.0, making scrypt an optional dependency and offering a pure JS version as a fallback.
The reasoning behind this is that scrypt is problematic to install in some systems, particularly those that don't have node-gyp setup and we have seen some weird issues when installing with elevated privileges (i.e. `sudo npm install -g scrypt`)
This avoids mistakenly placing process event handlers on the parent process
whenever the `embark-core` package is loaded. Also, don't listen for an
`'exit'` event on `process` and then call `process.exit(0)` since the event "is
emitted after the child process ends" ([docs][docs]), i.e. it's unnecessary to
do so and in any case it's not correct to always exit with code `0`.
[docs]: https://nodejs.org/docs/latest-v10.x/api/child_process.html#child_process_event_exit
dependabot-preview[bot]
Michael Bradley, Jr
Jonathan Rainville
Pascal Precht
André Medeiros
Iuri Matias
Eric Mastro
snyk-bot
Ricardo Guilherme Schmidt