Michael Bradley, Jr ee56f37713 build: implement collective typecheck
This PR replaces #2057.

Implement a collective typecheck action that can be invoked in the root of the
monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`.
Include the watch-mode typecheck action as part of `yarn start` (a.k.a
`yarn watch`).

To activate collective typecheck for a package in the monorepo, its
`package.json` file should specify:

```
{
  "embark-collective": {
    "typecheck": true
  }
}
```
*-or-*
```
{
  "embark-collective": {
    "typecheck": {...}
  }
}
```

Where `{...}` above is a `tsconfig.json` fragment that will be merged into the
config generated for the package according the same rules that `tsc` applies
when merging [configs][config].

When collective typecheck begins, it generates a `tsconfig.json` for the root
of the monorepo and for each package that is activated for the action. If the
generated JSON is different than what's on disk for the respective root/package
config, or if the config is not present on disk, then it will be
written. Changes to generated `tsconfig.json` files should be committed; such
changes will arise when there are structural changes to the monorepo, e.g. a
package is added, removed, moved and/or the directory containing it is
renamed. Since the configs are only generated at the beginning of collective
typecheck, when structural changes are made in the monorepo `yarn typecheck`
(or `yarn start` or `yarn watch:typecheck`) should be restarted.

Nearly all of the packages in the monorepo (i.e. all those for which it makes
sense) have been activated for collective typecheck. Even those packages that
don't contain `.ts` sources are activated because `tsc` can make better sense
of the code base as a whole owing to the project references included in the
generated `tsconfig.json` files. Also, owing to the fully cross-referenced
`tsconfig.json` files, it's possible for `tsc` to type check the whole code
base without babel (`yarn build` or `yarn watch:build`) having been run
beforehand.

**NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive:
on this author's 2019 MacBook Pro it takes around three minutes, the fans spin
up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has
completed, the next full typecheck will complete in a few seconds or less; and
when running in watch-mode there is likewise a *big* speedup once a full
typecheck has completed, whether that full check happened before it's running
in watch-mode or when watch-mode itself resulted in a full check before
switching automatically to incremental check, as well a corresponding *big*
reduction in resource consumption. A full check will be needed any time
`yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh
clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`.

The combination of options in each generated package-level `tsconfig.json` and
the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript
declaration files) into the `dist/` directory of each package. That
output is intended to live side-by-side with babel's output, and therefore the
`"rootDir"` option in each generated config is set to `"./src"`.

In projects activated for collective typecheck, `.js` may be converted to `.ts`
and/or `.ts` sources may be added without any additional changes needed in
package-level `package.json`.

---

Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into
`packages/core/core` (`embark-core`), refactor other packages' imports
accordingly, and delete `packages/core/typings` from the monorepo. This results
in some similarly named but incompatible types exported from `embark-core`
(e.g. `Events` and `EmbarkEvents`, the latter being the one from
`packages/core/typings`); future refactoring should consolidate those types. To
avoid circular dependency relationships it's also necessary to split out
`Engine` from `embark-core` into its own package (`embark-engine`) and to
introduce a bit of duplication, e.g. the `Maybe` type that's now defined in
both `embark-i18n` and `embark-core`.

In the process of the types reorg, move many dependencies spec'd in various
`package.json` to the `package.json` of the package/s that actually depend on
them, e.g. many are moved from `packages/embark/package.json` to
`packages/core/engine/package.json`. Related to those moves, fix some Node.js
`require`-logic related to bug-prone dependency resolution.

Fix all type errors that appeared as a result of activating collective
typecheck across the whole monorepo.

Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors.

Tidy up and add a few items in the root `package.json` scripts.

Bump lerna from `3.16.4` to `3.19.0`.

Bumpt typescript from `3.6.3` to `3.7.2`.

Bumpt tslint from `5.16.0` to `5.20.1`.

Make various changes related to packages' `import`/`require`ing packages that
weren't spec'd in their respective `package.json`. More refactoring is needed
in this regard, but changes were made as the problems were observed in the
process of authoring this PR.

[config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
2019-12-13 13:59:47 -05:00

307 lines
12 KiB
JavaScript

import { __ } from "embark-i18n";
const async = require("async");
// eslint-disable-next-line security/detect-child-process
const {spawn, exec} = require("child_process");
const path = require("path");
const constants = require("embark-core/constants");
const WhisperGethClient = require("./whisperGethClient.js");
import { IPC } from "embark-core";
import { compact, dappPath, defaultHost, dockerHostSwap, embarkPath} from "embark-utils";
const Logger = require("embark-logger");
// time between IPC connection attempts (in ms)
const IPC_CONNECT_INTERVAL = 2000;
class Blockchain {
/*eslint complexity: ["error", 50]*/
constructor(userConfig, clientClass, communicationConfig) {
this.userConfig = userConfig;
this.env = userConfig.env || "development";
this.isDev = userConfig.isDev;
this.onReadyCallback = userConfig.onReadyCallback || (() => {});
this.onExitCallback = userConfig.onExitCallback;
this.logger = userConfig.logger || new Logger({logLevel: "debug", context: constants.contexts.blockchain}); // do not pass in events as we don't want any log events emitted
this.events = userConfig.events;
this.isStandalone = userConfig.isStandalone;
this.certOptions = userConfig.certOptions;
let defaultWsApi = clientClass.DEFAULTS.WS_API;
if (this.isDev) defaultWsApi = clientClass.DEFAULTS.DEV_WS_API;
this.config = {
silent: this.userConfig.silent,
client: this.userConfig.client,
ethereumClientBin: this.userConfig.ethereumClientBin || this.userConfig.client,
networkType: this.userConfig.networkType || clientClass.DEFAULTS.NETWORK_TYPE,
networkId: this.userConfig.networkId || clientClass.DEFAULTS.NETWORK_ID,
genesisBlock: this.userConfig.genesisBlock || false,
datadir: this.userConfig.datadir,
mineWhenNeeded: this.userConfig.mineWhenNeeded || false,
rpcHost: dockerHostSwap(this.userConfig.rpcHost) || defaultHost,
rpcPort: this.userConfig.rpcPort || 8545,
rpcCorsDomain: this.userConfig.rpcCorsDomain || false,
rpcApi: this.userConfig.rpcApi || clientClass.DEFAULTS.RPC_API,
port: this.userConfig.port || 30303,
nodiscover: this.userConfig.nodiscover || false,
mine: this.userConfig.mine || false,
account: {},
whisper: (this.userConfig.whisper !== false),
maxpeers: ((this.userConfig.maxpeers === 0) ? 0 : (this.userConfig.maxpeers || 25)),
bootnodes: this.userConfig.bootnodes || "",
wsRPC: (this.userConfig.wsRPC !== false),
wsHost: dockerHostSwap(this.userConfig.wsHost) || defaultHost,
wsPort: this.userConfig.wsPort || 8546,
wsOrigins: this.userConfig.wsOrigins || false,
wsApi: this.userConfig.wsApi || defaultWsApi,
vmdebug: this.userConfig.vmdebug || false,
targetGasLimit: this.userConfig.targetGasLimit || false,
syncMode: this.userConfig.syncMode || this.userConfig.syncmode,
verbosity: this.userConfig.verbosity
};
this.devFunds = null;
if (this.userConfig.accounts) {
const nodeAccounts = this.userConfig.accounts.find((account) => account.nodeAccounts);
if (nodeAccounts) {
this.config.account = {
numAccounts: nodeAccounts.numAddresses || 1,
password: nodeAccounts.password,
balance: nodeAccounts.balance
};
}
}
if (this.userConfig.default || JSON.stringify(this.userConfig) === "{'client':'geth'}") {
if (this.env === "development") {
this.isDev = true;
} else {
this.config.genesisBlock = embarkPath("templates/boilerplate/config/privatenet/genesis.json");
}
this.config.datadir = dappPath(".embark/development/datadir");
this.config.wsOrigins = this.config.wsOrigins || "http://localhost:8000";
this.config.rpcCorsDomain = this.config.rpcCorsDomain || "http://localhost:8000";
this.config.targetGasLimit = 8000000;
}
this.config.account.devPassword = path.join(this.config.datadir, "devPassword");
const spaceMessage = "The path for %s in blockchain config contains spaces, please remove them";
if (this.config.datadir && this.config.datadir.indexOf(" ") > 0) {
this.logger.error(__(spaceMessage, "datadir"));
process.exit(1);
}
if (this.config.account.password && this.config.account.password.indexOf(" ") > 0) {
this.logger.error(__(spaceMessage, "accounts.password"));
process.exit(1);
}
if (this.config.genesisBlock && this.config.genesisBlock.indexOf(" ") > 0) {
this.logger.error(__(spaceMessage, "genesisBlock"));
process.exit(1);
}
this.client = new clientClass({config: this.config, env: this.env, isDev: this.isDev, communicationConfig: communicationConfig});
if (this.isStandalone) {
this.initStandaloneProcess();
}
}
/**
* Polls for a connection to an IPC server (generally this is set up
* in the Embark process). Once connected, any logs logged to the
* Logger will be shipped off to the IPC server. In the case of `embark
* run`, the BlockchainListener module is listening for these logs.
*
* @return {void}
*/
initStandaloneProcess() {
let logQueue = [];
// on every log logged in logger (say that 3x fast), send the log
// to the IPC serve listening (only if we're connected of course)
this.logger.events.on("log", (logLevel, message) => {
if (this.ipc.connected) {
this.ipc.request("blockchain:log", {logLevel, message});
} else {
logQueue.push({logLevel, message});
}
});
this.ipc = new IPC({ipcRole: "client"});
// Wait for an IPC server to start (ie `embark run`) by polling `.connect()`.
// Do not kill this interval as the IPC server may restart (ie restart
// `embark run` without restarting `embark blockchain`)
setInterval(() => {
if (!this.ipc.connected) {
this.ipc.connect(() => {
if (this.ipc.connected) {
logQueue.forEach((message) => {
this.ipc.request("blockchain:log", message);
});
logQueue = [];
this.ipc.client.on("process:blockchain:stop", () => {
this.kill();
process.exit(0);
});
}
});
}
}, IPC_CONNECT_INTERVAL);
}
runCommand(cmd, options, callback) {
this.logger.info(__("running: %s", cmd.underline).green);
if (this.config.silent) {
options.silent = true;
}
return exec(cmd, options, callback);
}
run() {
var self = this;
this.logger.info("===============================================================================".magenta);
this.logger.info("===============================================================================".magenta);
this.logger.info(__("Embark Whisper using %s", self.client.prettyName.underline).magenta);
this.logger.info("===============================================================================".magenta);
this.logger.info("===============================================================================".magenta);
if (self.client.name === constants.blockchain.clients.geth) this.checkPathLength();
let address = "";
async.waterfall([
function checkInstallation(next) {
self.isClientInstalled((err) => {
if (err) {
return next({ message: err });
}
next();
});
},
function getMainCommand(next) {
self.client.mainCommand(address, function (cmd, args) {
next(null, cmd, args);
}, true);
}
], function(err, cmd, args) {
if (err) {
self.logger.error(err.message);
return;
}
args = compact(args);
let full_cmd = cmd + " " + args.join(" ");
self.logger.info(__(">>>>>>>>>>>>>>>> running: %s", full_cmd.underline).green);
self.child = spawn(cmd, args, {cwd: process.cwd()});
self.child.on("error", (err) => {
err = err.toString();
self.logger.error("Blockchain error: ", err);
if (self.env === "development" && err.indexOf("Failed to unlock") > 0) {
self.logger.error("\n" + __("Development blockchain has changed to use the --dev option.").yellow);
self.logger.error(__("You can reset your workspace to fix the problem with").yellow + " embark reset".cyan);
self.logger.error(__("Otherwise, you can change your data directory in blockchain.json (datadir)").yellow);
}
});
// TOCHECK I don't understand why stderr and stdout are reverted.
// This happens with Geth and Parity, so it does not seems a client problem
self.child.stdout.on("data", (data) => {
self.logger.error(`${self.client.name} error: ${data}`);
});
self.child.stderr.on("data", async (data) => {
data = data.toString();
if (!self.readyCalled && self.client.isReady(data)) {
self.readyCalled = true;
self.readyCallback();
}
self.logger.info(`${self.client.name}: ${data}`);
});
self.child.on("exit", (code) => {
let strCode;
if (code) {
strCode = "with error code " + code;
} else {
strCode = "with no error code (manually killed?)";
}
self.logger.error(self.client.name + " exited " + strCode);
if (self.onExitCallback) {
self.onExitCallback();
}
});
self.child.on("uncaughtException", (err) => {
self.logger.error("Uncaught " + self.client.name + " exception", err);
if (self.onExitCallback) {
self.onExitCallback();
}
});
});
}
readyCallback() {
if (this.onReadyCallback) {
this.onReadyCallback();
}
}
kill() {
if (this.child) {
this.child.kill();
}
}
checkPathLength() {
let _dappPath = dappPath("");
if (_dappPath.length > 66) {
// this.logger.error is captured and sent to the console output regardless of silent setting
this.logger.error("===============================================================================".yellow);
this.logger.error("===========> ".yellow + __("WARNING! ÐApp path length is too long: ").yellow + _dappPath.yellow);
this.logger.error("===========> ".yellow + __("This is known to cause issues with starting geth, please consider reducing your ÐApp path\'s length to 66 characters or less.").yellow);
this.logger.error("===============================================================================".yellow);
}
}
isClientInstalled(callback) {
let versionCmd = this.client.determineVersionCommand();
this.runCommand(versionCmd, {}, (err, stdout, stderr) => {
if (err || !stdout || stderr.indexOf("not found") >= 0 || stdout.indexOf("not found") >= 0) {
return callback(__("Ethereum client bin not found:") + " " + this.client.getBinaryPath());
}
const parsedVersion = this.client.parseVersion(stdout);
const supported = this.client.isSupportedVersion(parsedVersion);
if (supported === undefined) {
this.logger.warn((__("WARNING! Ethereum client version could not be determined or compared with version range") + " " + this.client.versSupported + __(", for best results please use a supported version")));
} else if (!supported) {
this.logger.warn((__("WARNING! Ethereum client version unsupported, for best results please use a version in range") + " " + this.client.versSupported));
}
callback();
});
}
}
export class BlockchainClient extends Blockchain {
constructor(userConfig, options, communicationConfig) {
if ((JSON.stringify(userConfig) === "{'enabled':true}") && options.env !== "development") {
options.logger.info("===> " + __("warning: running default config on a non-development environment"));
}
// if client is not set in preferences, default is geth
if (!userConfig.client) userConfig.client = constants.blockchain.clients.geth;
// if clientName is set, it overrides preferences
if (options.clientName) userConfig.client = options.clientName;
userConfig.isDev = (userConfig.isDev || userConfig.default);
userConfig.env = options.env;
userConfig.onReadyCallback = options.onReadyCallback;
userConfig.onExitCallback = options.onExitCallback;
userConfig.logger = options.logger;
userConfig.certOptions = options.certOptions;
userConfig.isStandalone = options.isStandalone;
super(userConfig, WhisperGethClient, communicationConfig);
}
}