## Problem
Doing read, then write each a trasaction or call exectues could get
heavy, especially with regular txs on
This was especially true in the tests, which led to deactivate the
tx logger in the tests
## Solution
Instead of reading the whole file, adding the JSON line a writing,
we instead just append some pseudo JSON to it that later gets read
and parsed correctly back to JSON
This commit adds two new configuration settings for Smart Contract configuration:
- `interfaces` - Any Smart Contract that represent an interface or is used for inheritance
- `libraries` - Any Smart Contract that is used as a library
This makes the configuration less redundant in cases where otherwise the `deploy`
property has been set to `false`, such as:
```
deploy: {
Ownable: {
deploy: false
},
...
}
```
The above can now be done via:
```
interfaces: ['Ownable'],
deploy: {
...
}
```
This commit introduces a new `global.getEvmVersion()` that can be used to
conditionally run tests, such as when tests rely on RPC APIs that are only
available in specific evm nodes.
## Issue
Transaction logs for contracts that were unknown to Embark (ie not in the dapp) would often log super large objects (filling the terminal) that were not formatted with spaces so were hard to read without actually be that useful. In addition, occasionally the object logged would throw the error `TypeError: Converting circular structure to JSON`.
## Fix
Set the log level for transaction logs that are not from a known contract to `debug`, so that they do not flood the terminal with often usused information.
Use `util.inspect` to print the transaction log instead of `JSON.stringify` to prevent circular structure errors.
If using `embark-snark` in the dapp as a plugin, and the dapp didn’t have any contracts, the `embark-snark` plugin would not simply exit early and essentially do nothing.
Fix `embark-snark` plugin so that it will run its routine regardless of whether or not the dapp has contracts or not.
Bump `circom` and `snarkjs` to their latest patch version.
## Issue 1 - “register” section missing
When the “register” section of `namesystem.json` was missing, ENS would not deploy the ENS contracts nor create the contracts’ artifacts.
### Fix 1
Fix ENS deployment routine to always deploy the ENS contracts. In the case of testnet/mainnet, the contracts’ addresses will be known and therefore will be understood as “already deployed” by the contract deployer.
## Issue 2 - “register” section exists for non-dev environment
Additionally, if a root domain was specified in the “register” section and the DApp connected to an external node where we do not own the ENS contracts (ie testnet), attempting to register a root domain would not be possible as we do not own the ENS contracts.
### Fix 2
Fix ENS deployment routine to check if we are on a network in which we own the ENS contracts. If we are not, and we have specified a “register” section, print a warning to the user that the registration will be ignored. Additionally, remove the “register” section.
The transaction-logger was slowing down tests because each Tx, it
would read a file and write to it, but that's slow and even got
slower as the file grew bigger
To fix, I removed the Tx-logger from text, along with the profiler
as they are useless for tests
(cherry picked from commit a656eea7dda83ad970615cffd51cce5d53a1d034)
There was a race condition in which the coverage module tried to read a generated
coverage report before it was actually generated.
The issue was that the coverage generation was done on Embark's `tests:finished` event
in a fire and forget manner via `emit('tests:finished')` which doesn't
ensure control flow.
This commit changes it to use `runActionsForEvent('tests:finished')` instead and also
registers the handler for coverage report generation as action via `registerActionForEvent()`.
This ensures that those actions are run first, before the code moves on with other
operations that might rely on the result of any of those actions.
Refactor typings as necessary. In order for `bignumber.js` in the root
`node_modules` to not conflict with `@types/bignumber.js`, specify
`"bignumber.js": "5.0.0"` in `devDependencies` and `"embark-ui/bignumber.js"`
in `nohoist` of the root `package.json`.
In `packages/plugins/rpc-manager` switch from callback to promise usage with
respect to `web3.eth.accounts.signTransaction` because of a [bug][bug]
discovered in web3 v1.2.4.
In `packages/plugins/solidity-tests` specialize the tsc compiler options with
`"skipLibCheck": true` because of a problematic web3-related typing in the
`.d.ts` files of the remix-tests package.
Bump ganache-cli from 6.4.3 to 6.7.0 (latest) because 6.4.3 doesn't support
`eth_chainId` but web3 1.2.4 makes use of the `eth_chainId` RPC method (EIP
695).
BREAKING CHANGE: bump embark's minimum supported version of parity from
`>=2.0.0` to `>=2.2.1`. This is necessary since web3 1.2.4 makes use of the
`eth_chainId` RPC method (EIP 695) and that parity version is the earliest one
to implement it.
[bug]: https://github.com/ethereum/web3.js/issues/3283
Was caused by the fact that each rpc-modifier got its own node
accounts, but once the eth_accounts modifier is enabled, the
"node" accounts are always going to contain the custom accounts too
Now, the node accounts are set once eth_accounts is initiated, so
that every modifier is on the same page
Also implement a second layer of protection to make sure that if a
duplication happens, it is gotten rid of
This PR replaces #2057.
Implement a collective typecheck action that can be invoked in the root of the
monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`.
Include the watch-mode typecheck action as part of `yarn start` (a.k.a
`yarn watch`).
To activate collective typecheck for a package in the monorepo, its
`package.json` file should specify:
```
{
"embark-collective": {
"typecheck": true
}
}
```
*-or-*
```
{
"embark-collective": {
"typecheck": {...}
}
}
```
Where `{...}` above is a `tsconfig.json` fragment that will be merged into the
config generated for the package according the same rules that `tsc` applies
when merging [configs][config].
When collective typecheck begins, it generates a `tsconfig.json` for the root
of the monorepo and for each package that is activated for the action. If the
generated JSON is different than what's on disk for the respective root/package
config, or if the config is not present on disk, then it will be
written. Changes to generated `tsconfig.json` files should be committed; such
changes will arise when there are structural changes to the monorepo, e.g. a
package is added, removed, moved and/or the directory containing it is
renamed. Since the configs are only generated at the beginning of collective
typecheck, when structural changes are made in the monorepo `yarn typecheck`
(or `yarn start` or `yarn watch:typecheck`) should be restarted.
Nearly all of the packages in the monorepo (i.e. all those for which it makes
sense) have been activated for collective typecheck. Even those packages that
don't contain `.ts` sources are activated because `tsc` can make better sense
of the code base as a whole owing to the project references included in the
generated `tsconfig.json` files. Also, owing to the fully cross-referenced
`tsconfig.json` files, it's possible for `tsc` to type check the whole code
base without babel (`yarn build` or `yarn watch:build`) having been run
beforehand.
**NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive:
on this author's 2019 MacBook Pro it takes around three minutes, the fans spin
up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has
completed, the next full typecheck will complete in a few seconds or less; and
when running in watch-mode there is likewise a *big* speedup once a full
typecheck has completed, whether that full check happened before it's running
in watch-mode or when watch-mode itself resulted in a full check before
switching automatically to incremental check, as well a corresponding *big*
reduction in resource consumption. A full check will be needed any time
`yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh
clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`.
The combination of options in each generated package-level `tsconfig.json` and
the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript
declaration files) into the `dist/` directory of each package. That
output is intended to live side-by-side with babel's output, and therefore the
`"rootDir"` option in each generated config is set to `"./src"`.
In projects activated for collective typecheck, `.js` may be converted to `.ts`
and/or `.ts` sources may be added without any additional changes needed in
package-level `package.json`.
---
Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into
`packages/core/core` (`embark-core`), refactor other packages' imports
accordingly, and delete `packages/core/typings` from the monorepo. This results
in some similarly named but incompatible types exported from `embark-core`
(e.g. `Events` and `EmbarkEvents`, the latter being the one from
`packages/core/typings`); future refactoring should consolidate those types. To
avoid circular dependency relationships it's also necessary to split out
`Engine` from `embark-core` into its own package (`embark-engine`) and to
introduce a bit of duplication, e.g. the `Maybe` type that's now defined in
both `embark-i18n` and `embark-core`.
In the process of the types reorg, move many dependencies spec'd in various
`package.json` to the `package.json` of the package/s that actually depend on
them, e.g. many are moved from `packages/embark/package.json` to
`packages/core/engine/package.json`. Related to those moves, fix some Node.js
`require`-logic related to bug-prone dependency resolution.
Fix all type errors that appeared as a result of activating collective
typecheck across the whole monorepo.
Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors.
Tidy up and add a few items in the root `package.json` scripts.
Bump lerna from `3.16.4` to `3.19.0`.
Bumpt typescript from `3.6.3` to `3.7.2`.
Bumpt tslint from `5.16.0` to `5.20.1`.
Make various changes related to packages' `import`/`require`ing packages that
weren't spec'd in their respective `package.json`. More refactoring is needed
in this regard, but changes were made as the problems were observed in the
process of authoring this PR.
[config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
This was caused by the fact that we registered the action to change
the accounts, but that action needs the node accounts, which is
a request to eth_accounts, hence creating a loop
The transaction logger takes care of intercepting any responses coming from the
blockchain proxy, determines what response it's dealing with and then extracts
information from the response to output useful log information to the user, such as
when a contract methods has been called etc.
It turned out that it didn't properly handle cases where value transactions
from account to account (such as done here in [Status Teller](a5ab4d4b26/embarkConfig/data.js (L100-L110))) have been made.
Since Embark couldn't map any of those accounts to actual contracts, while
still having stored corresponding transactions, it logged them as transactions
from "Unknown Smart Contracts".
This commit ensures via heuristics that, if the address a transactions is sent to is
included in any of the nodes accounts, the transaction logger logs a useful message
that a certain value is sent from one to anover account.
fix(@embark/geth): only register console command if in dev mode; use endpoint; use dev account for regular txs that fix geths stuck tx issue
fix(@embark/geth): only do regular txs if miningMode is dev
fix(@embark/geth): only register console command if in dev mode; use endpoint
fix(@embark/geth): only register console command if in dev mode; use endpoint
fix(@embark/geth): only register console command if in dev mode; use endpoint
fix(@embark/geth): only register console command if in dev mode; use endpoint
It's a relevant `package.json` if the package has a test suite. If `test` isn't
specified then the tests won't run in CI (they have't been running).
Include a fix for one failing test in `packages/plugins/ens` and two failing
test in `packages/stack/deployment`. In all three cases, the mock embark
object's config needed a `blockchainConfig` section with `{ enabled: true }`.
As part of 8ed5376533 we made `ENS.getEnsConfig()`
asynchronous, hence it needs a callback. This change was not done in one of the
ENS tests, resulting in the test to fail. The test mocked that particular API,
turning it into noop, which is why the test kept timing out.
This commit fixes the test by not mocking the API and adding the needed configuration
to the environment.
Prior to this commit, registering ENS subdomains would fail due to `defaultAccount`
being null. This is because the underlying API was still relying on `blockchain:defaultAccount:get`.
However since the bigger refactor, every plugin/module is in charge of creating its own
blockchain connector instance and ensuring it has a default account.
This commit makes use of ENS' module's `webDefaultAccount` async getter to
ensure a valid default account is set when registering an ENS subdomain.
Add back DevTxs.
DevTxs sends a zero-value tx to push through any pending txs that may get stuck with geth while in `—dev` mode.
When the geth client is running, devtxs is started automatically, sending a transaction every 10 seconds.
Converted the legacy code in to typescript.
Re-enable debugger for cockpit and console.
Add missing request `blockchain:getTransaction` required to start a debugging session.
Fix error in console when stopping a debug session.
Add option in communication config to choose which Whisper client to use.
Because Parity’s implementation of Whisper is not compatible with Whisper v6, and therefore web3.js in its current form, the following changes have been made:
1. remove any functionality associated with launching a Parity Whisper process.
2. Warn the user when the Parity Whisper client has been opted for in the communication config.
3. Return an error for API calls when Parity Whisper client has been opted for in the communication config.
4. Update Cockpit’s Communication module to show errors returned from API calls.
5. Update the messaging configuration documentation for the new communication client option.
Users can turn of blockchain support if they want to using the blockchain.js
configuration. In practice however, this has never properly worked as several
places in Embark's codebase weren't actually honoring that configuration value.
This commit introduces the necessary changes so that disabling blockchain support will:
No longer generate blockchain related EmbarkJS artifacts
No longer try to deploy Smart Contracts but still compile them
This commit removes the need for `EmbarkJS.onReady()` and `EmbarkJS.Blockchain.blockchainConnector` APIs
in the ENS provider implementation and instead relies purely on vanilla `Web3`. This comes
with the effect that `EmbarkJS.Names` needs to figure out itself what to connect
to, as well as when a connection has been established.
To make that possible, `EmbarkJS.Names` now implements a similar algorithm to
`EmbarkJS.Blockchain` that tries to connect different endpoint, given a `dappConnection`
configuration.
If no `dappConnection` configuration is given via `namesystem.json`, Embark will fallback
to the same connection list that's provided in `contracts.js|json`.
wip
1. get rid of the whisper layer and handle everything in the communication layer
2. Create a gethWhisper and a parityWhisper plugin that has the same files as packages/embark/src/lib/modules/geth, except with a whisperClient instead of gethClient, and will include most of https://github.com/embark-framework/embark/blob/master/packages/plugins/whisper/src/index.js.
3. Get rid of any whisper registration in geth.
4. In the whisperGeth and whisperParity plugins, modify the request for communication:node:register, to end up calling `startWhisperNode`
When whisper was being started in a new process, the message displayed in the console was “Starting Blockchain node in another process”. This is misleading as it is the same message that geth/parity use to start up their processes.
Update whisper initialisation process messaging so it states “Starting Whisper node in another process”.
When `eth_unsubscribe` is received in the proxy, ensure this request is forwarded through on the correct socket (the same socket that was used for the corresponding `eth_subscribe`).
Move subscription handling for `eth_subscribe` and `eth_unsubscribe` to RpcModifiers (in `rpc-manager` package).
For each `eth_subscribe` request, a new `RequestManager` is created. Since the endpoint property on the proxy class was updated to be a provider, the same provider was being assigned to each new `RequestManager` and thus creating multiple event handlers for each subscription created. To circumvent this, we are now creating a new provider for each `RequestManager`.
Co-authored-by: Pascal Precht <pascal.precht@googlemail.com>
This PR adds support for EIP-712 (https://github.com/ethereum/EIPs/blob/master/EIPS/eip-712.md), by allowing the signing of typed data transactions using the `eth_signTypeData_v3` or `eth_signTypedData` request.
Add a module called `embark-rpc-manager` to handle all RPC transaction modifications.
The internal events management module extracted to `embark-core`. This allows other modules (ie plugins) to use a private instance of Embark’s event API.
Remove transaction modifications in `embark-accounts-manager` and leave only the functionality in this module that handle account management (ie funding of accounts). Other functionality was moved to `embark-rpc-manager`.
- Transactions should now reflect when a new node account is added via `personal_newAccount` or via the node itself (and once `eth_accounts` is requested).
- In the proxy, errors are handled for all cases and are now JSON-RPC spec-compliant.
- Always register `eth_signTypedData` RPC response and display error message if account cannot be found.
NOTE: Updated yarn.lock due to conflict after rebase.
Avoid race condition with rpc modifications
Refactor proxy to make request handling asynchronous, and to allow circumvention of forwarding requests to the node in cases where it is not needed.
Move loading of plugins to after core and stack components (in cmd_controller).
We should load custom plugins LAST (ie after all core and stack components, and after all Embark plugins). To do this, we disable automatic loading of plugins in the config, and allow them to be loading explicitly in cmd_controller.
Create a system that allows registration of custom plugins. Each plugin is generated in EmbarkJS by instantiating the registered class. For example, we can register a Plasma plugin and it will be generated in to EmbarkJS.Plasma with an instantiation of `embarkjs-plasma`. The DApp itself can then do what it wants with `EmbarkJS.Plasma` (ie call `EmbarkJS.Plasma.init()`.
NOTE: loading of custom plugins needs to be applied to all other commands in `cmd_controller`. This PR only loads custom plugins for the `run` command.
NOTE: This PR is based on branch fix/ws-eth-subscribe (#1850). I will rebase this PR when (and if) #1850 is merged.
BREAKING CHANGE:
Related to #1985. Prior to Embark's minimum supported version of Node.js being
bumped to to 10.17.0, Embark was incompatible with any relatively recent
release of the `ipfs-http-client` package.
While *internal* changes are needed re: ipfs's `Buffer` export for
e.g. `embark_demo` to function correctly *(and this PR makes those changes)*,
Embark otherwise runs/tests okay.
Keep in mind #2033.
However, if a dApp author were to explicitly `require('ifps-api')` in the
front-end that wouldn't work as before; and swapping `"ipfs-http-client"` for
`"ipfs-api"` might also not be enough — there are API changes that dApp authors
would need to consider, regardless of Embark presently supplying the dependency
and EmbarkJS wrapping around it.
Closes#1994.
Hook `embark-profiler` into the engine's contracts group.
Adjust `request` signatures to match v5 usage.
Ensure gas numbers are formatted consistently in the ascii table.
Display non-json output in cockpit's console as monospace/pre to preserve
profiler table formatting.
The blockchain module should not contain any ethereum-specific code, and currently it contains a check to see if the blockchain has already been started.
This PR moves this check to blockchain client (ie geth or parity). This check function is registered along with the started callback.
style(@embark/stack/blockchain): add missing semicolon
Turns out that, when Embark tries to replace ENS subdomains, it would fail
if `subdomains` aren't defined in the namesystem configuration.
This commit adds a safeguard so that not defining `subdomains` is fine.
This commit fixes the issue that it wasn't possible anymore to use named constructor arguments
in Smart Contract configurations.
For example, the following Smart Contract expects two constructor arguments:
```solidity
contract SomeContract {
constructor(address[] _addresses, int initialValue) public {}
}
```
The first being a list of addresses, the second one a number. This can be configured as:
```js
SomeContract: {
args: [
["$MyToken2", "$SimpleStorage"],
123
]
}
```
Notice how the order of arguments matters here. `_addresses` come first in the constructor,
so they have to be defined first in the configuration as well.
Another way to configure this is using named arguments, which is what's broken prior to this commit:
```js
SomeContract: {
args: {
initialValue: 123,
_addresses: ["$MyToken2", "$SimpleStorage"]
}
}
```
Using a notation like this ^ the order no longer matters as Embark will figure out the right
values for the constructor arguments by their names.
The reason this is broken is because there are several modules in Embark that register and
run a `deployment:contract:beforeDeploy` action, which are allowed to mutate this configuration
object. One of those modules is the `ens` module, which searches for ENS names in the arguments
and figure out whether it has to replace it with a resolved address.
One thing that particular module didn't take into account is that `args` could be either and
array, or an object and will always return an array, changing the shape of `args` in case it was
an object.
This is a problem because another module, `ethereum-blockchain-client`, another action is registered
that takes this mutated object in `determineArguments()` and ensure that, if `args` is actually an
object, the values are put in the correct position matching the constructor of the Smart Contract in
question.
One way to solve this was to use the newly introduced [priority](https://github.com/embark-framework/embark/pull/2031) and ensure
that `ens` action is executed after `ethereum-blockchain-client`'s.
However, the actual bug here is that the ENS module changes the shape of `args` in the first place,
so this commit ensures that it preserves it.
The `WhisperGethClient` returns empty strings for its "new account" and "list
accounts" commands; if a command is an empty string then `Blockchain` should
not execute it.
* fix(@embark/solidity): fix solidity ipc connection with blockchain
When blockchain was run in another process, the IPC was connected,
but the compiler was not loaded, so the IPC calls never returned
* fix(@embark/geth): fix cb is not a fn because it needs request2
We've introduced a regression in https://github.com/embark-framework/embark/commit/f9557d4c9 where invalid data
has been sent to web3's `ecRecover()` method to verify signed messages.
This causes an internal server error and the utility feature inside Cockpit
unsuable.
This commit ensures that the correct data is sent to `ecRecover()` making verifying
messages through Cockpit functional again.
When making use of the `useBuiltIns: 'usage'` option for @babel/preset-env
(which is the case for all transpiled packages in Embark's monorepo) a package
needs to have both @babel/runtime-corejs3 and core-js@3 specified as
dependencies.
* build(deps): move @types/os-locale from packages/embark to packages/core/i18n
* build(deps) move @types/globule from packages/embark to packages/plugins/coverage
* build(deps): refactor stack/{api,proxy,webserver} deps relative to packages/embark
* build(deps): remove unneeded @types/async dep from packages/stack/test-runner
* build(deps): remove unneeded deps from packages/embark
BREAKING CHANGE:
node: >=10.17.0 <12.0.0
npm: >=6.11.3
yarn: >=1.19.1
node v10.17.0 is the latest in the 10.x series and is still in the Active LTS
lifecycle. Embark is still not compatible with node's 12.x and 13.x
series (because of some dependencies), otherwise it would probably make sense
to bump our minimum supported node version all the way to the most recent 12.x
release.
npm v6.11.3 is the version that's bundled with node v10.17.0.
yarn v1.19.1 is the most recent version as of the time node v10.17.0 was
released.
* refactor(@embark/dapps/tests/app): use function syntax
These changes don't fix the race conditions related to the test dapp's tests
but are a step in the right direction.
* refactor(@embark/dapps/tests/contracts): adjustments to get tests passing
Further refactoring is needed re: potentially duplicated or overlapping logic
in `packages/plugins/solidity-tests` and
`packages/core/utils/src/solidity/remapImports.ts`, as well in disabled test
dapp tests
* test(dapps/tests/app): temporarily disable intermittently failing tests
They are failing because of a race condition; once that race condition has been
fixed these tests should be reenabled.
* fix(@embark/solidity-tests): fix importing the library for the tests
* build(@embark/stack/blockchain-client): remove unneeded typescript related scripts and deps
In addition to being unneeded their presence is causing build errors.
* build(embark): remove unneeded typescript related scripts and deps
In addition to being unneeded their presence is causing build errors.
* build(@embark/plugins/ethereum-blockchain-client): remove unneeded typescript related scripts, deps
In addition to being unneeded their presence is causing build errors.
* build(@embark/plugins/ganache): remove unneeded typescript related scripts and deps
In addition to being unneeded their presence is causing build errors.
* build(@embark/plugins/geth): remove unneeded typescript related scripts and deps
In addition to being unneeded their presence is causing build errors.
* fix(@embark/plugins/transaction-logger): require 'web3' not 'Web3'
* fix(@embark/utils/solo): spawn npm(.cmd) instead of npx(.cmd)
* test(@embark/plugins/basic-pipeline): add test stub
* test(@embark/stack/blockchain): add test stub
Jonathan Rainville
Iuri Matias
Pascal Precht
emizzle
Michael Bradley, Jr
dependabot-preview[bot]