Just a more sensible name since it's not a homepage of any sort.
Test plan: Flow passes, `yarn build2` and `yarn start2` both succeed.
Also: `git grep homepage2` gets no hits.
Now you pass `--instance PATH` rather than `--instance=PATH` which is
agreed to be much much better. (It tab completes.)
Test plan: Do the obvious thing. :)
Per discussion with @hammadj, @topocount, and @wchargin, we are planning
to have the frontend2 system use react-admin at the top level. Per
investigation by @topocount, react-admin conflicts with the older
version of react-router that we use.
As such, this commit wildly simplifies the homepage2 system so we no
longer have any routing, and instead we just statically render the
index.html file. We also removed the `Assets` type, not because we are
sure we don't need it, but because we didn't want to debug it while we
were all pairing. @wchargin offered to fix it up later.
Test plan:
- run `yarn start2 --instance=PATH` and observe that the "Under
Construction" message displays, along with console messages showing that
data loaded successfully.
- run `yarn build2` and copy files from `build2` into the root of a cli2
instance. Run an http server in that instance, and observe that the
frontend displays properly per instructions above.
Paired with: @wchargin
Paired with: @hammadj
Paired with: @topocount
This modifies the frontend2 so that we can load real data from cli2
instances, including:
- the root `sourcecred.json` config
- files inside the `config/` directory
- files inside the `output/` directory
This works both for the dev server and the compiled output.
In the case of the dev server, it's now necessary to provide the path to
the cli2 instance you're developing against. You can pass this via the
`--instance` argument, as in `yarn start2 --instance=/path/`, or via
the `$SOURCECRED_DEV_INSTANCE` environment variable (recommended). If
neither is provided, or if that path doesn't look like a valid instance,
an error is thrown.
In the case of the built output, given a valid sc2 instance, you can set
it up via the following:
```
cp -r build2/favicon.png build2/index.html build2/static $INSTANCE
```
Then spin up a simple http server in the $INSTANCE.
You can look at console messages in the frontend to verify that the
instance is working (note it expects to see a discourse config file, not
a GitHub config file as was used in some previous examples).
Test plan:
Setup an example Discourse instance, and then turn on the server via
`yarn start2 --instance=PATH`,
`SOURCECRED_DEV_INSTANCE=path yarn start2`,
and by manually copying in the built outputs using the instructions
above.
In each case, you can load the homepage and check the console to see
that assets loaded successfully.
The frontend build system has a bunch of logic for loading in the list
of projectIds and including them in the build. Fortunately, in
frontend2, we won't need this. This commit simply removes all that
logic.
Test plan: Grepping for `projectIds` finds no hits inside the frontend2
system, except for a comment noting that we'll be able to remove an
argument once the transition is complete. `yarn start2` still works as
expected.
The cli2 ("instance") system has a foundationally different assumption
about how the frontend works: rather than having a unified frontend that
abstracts over many separate SourceCred projects, we'll have a single
frontend entry per instance. This means we no longer need (for example)
to make project IDs available at build time.
Our frontend setup and server side rendering is pretty complex, so
rather than rebuild it from scratch, I'm going to fork it into an
independent copy and then change it to suit our needs.
To start here, I've duplicated the `src/homepage` directory into
`src/homepage2`, duplicated the webpack config to
`config/webpack.config.web2.js`, and duplicated the paths and
package.json scripts.
Test plan:
Run `yarn start2` and it will start an identical frontend, using the
duplicated directory. Run `yarn build2` and it will build a viable
frontend into the `build2` directory. `build2` is gitignored.
Summary:
This patch creates a new binary, `./bin/sc2`, which will be the home for
a rewrite of the CLI intended to implement an instance system. See:
<https://discourse.sourcecred.io/t/sourcecred-instance-system/244>
Paired with @decentralion.
Test Plan:
Run `yarn backend && node ./bin/sc2.js`, which should nicely fail with a
“not yet implemented” message.
wchargin-branch: cli2-skeleton
Summary:
Generated by running `yarn prettify`.
Test Plan:
Running `yarn check-pretty` no longer prints any errors.
wchargin-branch: prettier-2.0.1-reformat
Summary:
This re-packages the build for the internal APIs exposed under #1526 to
be more browser-friendly. Removing `target: "node"` (and adding an
explicit `globalObject: "this"` for best-effort cross-compatibility) is
the biggest change from the backend build; removing all the extra
loaders and static site generation is the biggest change from the
frontend build.
This build configuration is forked from `webpack.config.backend.js`.
Test Plan:
Run `yarn api`, then upload the contents of `dist/api.js` to an
Observable notebook and require it as an ES module. Verify that the
SourceCred APIs are exposed: e.g., `sourcecred.core.graph.Graph` should
be a valid constructor.
wchargin-branch: api-build
Summary:
For convenient import by scripts and Observable notebooks that want to
use SourceCred code outside its normal build system. We export a subset
of the codebase, including some core data structures and algorithms and
also some plugin metadata, but no plugin loading code.
To build, run `yarn backend` (or `yarn backend --watch`), then grab the
new `bin/api.js` file.
Test Plan:
Sample usage, with normal Node:
```javascript
const {
core: {
graph: {Graph, NodeAddress, EdgeAddress},
},
} = require("./api").default;
function node(address) {
return {
address,
description: "blurgh",
timestampMs: -1,
};
}
const g = new Graph();
g.addNode(node(NodeAddress.fromParts(["people", "alice"])));
g.addNode(node(NodeAddress.fromParts(["people", "bob"])));
g.addEdge({
address: EdgeAddress.fromParts(["friendship"]),
src: NodeAddress.fromParts(["people", "alice"]),
dst: NodeAddress.fromParts(["people", "bob"]),
timestampMs: 0,
});
console.log(require("json-stable-stringify")(g));
```
This prints a valid graph JSON object.
wchargin-branch: api-bundle
Before we added the concept of "SourceCred Projects", we tracked cred
instances via their GitHub repostiory id. The replacement for this
system was added in #1238, I missed the RepoIdRegistry in the cleanup.
This commit removes all code pertaining to the now-obsolete
RepoIdRegistry.
Test plan:
- `yarn test --full` passes
- manual inspection of `yarn start`; it still loads properly
- manual inspection of the output for build_static_site.sh
- `git grep repoIdRegistry` returns no hits
Summary:
PRs created from forks don’t have credentials when running CI. This
commit causes the `test-full` job (which requires credentials) to fail
fast with a helpful error message.
Test Plan:
Push distinct versions of this commit to a fork and to the main
repository, and open pull requests for each. Note that the tests pass
from the main repository, but fail with a nice message from the fork:
![Screenshot of expected fast-fail behavior][ss]
The “team member pushes to trusted branch” workflow has already been
successfully exercised for #1521.
[ss]: https://user-images.githubusercontent.com/4317806/71707839-b782ab00-2da1-11ea-8aa9-7d8720538a87.png
wchargin-branch: forked-pr-fail-fast
This commit swaps usage over to the new implementation of `cli/load`
(the one that wraps `api/load`) and makes changes throughout the project
to accomodate that we now track instances by Project rather than by
RepoId.
Test plan: Unit tests updated; run `yarn test --full`. Also, for safety:
actually load a project (by whole org, why not) and verify that the
frontend still works.
I'm re-organizing SC data to be oriented on the graph, rather than on
plugin-specific data structures. So there is no longer a need for the
git loading logic which orients around saving a repository.json file
that's been potentially merged across repos, or indeed the logic for
merging repositories at all. So I'm removing `git/loadGitData`,
`git/mergeRepository`, and relatives.
Test plan: `yarn test --full` passes.
See #1243 for context. This is basically a more aggressive version of
pull #1230 -- instead of just running unit tests in isolation, we also
run flow in isolation, and kill the servers afterwards.
Test plan: See how this fares in CI :)
Ever since I upgraded all of the dependencies, we've been having
regular CI failures, which seem to share a common root cause of memory
exhaustion. Here are some examples: [1], [2].
[1]: https://circleci.com/gh/sourcecred/sourcecred/1246
[2]: https://circleci.com/gh/sourcecred/sourcecred/1239
After some experimentation, I've found that we can solve the
issue by ensuring that jest runs on its own in CI, so that it doesn't
contend with other tests for memories. Also, I reduce its max workers to
2, which matches the number of CPUs in the CircleCI containers.
Unfortunately, this does increase our build time. The postcommit (non
full) test now takes 45-60s (up from 30-50s), and the full test is also
a little slower. However, building in about one minute is still
acceptably fast, and having regular flakey test failures is not
acceptable, so this is still a win.
If we want to improve on this in the future, we should look into the git
shells getting spawned in `config/env.js`. I noticed that they were
often involved in the out-of-memory failures.
Also, I modified `.circleci/config.yml` so that any branch matching the
regular expression `/ci-.*/` will trigger a full build. That makes it
easier to test against CI failures.
Test plan: I ran about ~10 full builds with this change, and more with
similar variations, and they all passed. Verify that the full builds
that are run for this commit also all pass! Also, verify that running
yarn test locally has unchanged behavior, and running
`yarn test --ci` locally lets jest run to completion before running
any other test.
Now that babel is upgraded, upgrading webpack was pretty
straightforward.
- We take advantage of the new `mode` config option, and no longer need
to manually set up Uglify plugin
- Uglifyjs is back, I checked the prod build output: it's very ugly
- I updated the RemoveBuildDirectoryPlugin per instructions, and
verified it still works.
- I verified that of `yarn backend`, `yarn build`, and `yarn start` all
still work as expected.
I moved `config/babel.js` to `.babelrc.js` because it seemed like babel
7 really wanted that. I also blew away our (complicated, copied from
create-react-app) config and replaced it with a much, much simpler one.
Test plan: `yarn test` passes, `yarn start` still serves a working
server, and `scripts/build_static_site.sh` still produces a working
site.
Possibly we lost some nice features re: React debugging; if so I'll add
them back as I miss them.
We have an old version of uglify, and it's causing problems with
compiling d3-array, and also when upgrading to a newer babel.
I'm going to disable it for now, then upgrade babel, and then upgrade
webpack. Upgrading webpack will get us a later version of uglify without
these issues.
This necessitated a number of type fixes:
- Upgraded the express flow-typed file to latest
- Added manual flow error suppression to where the express flow-typed
file is still using a deprecated utility type
- Removed type polymorphism support on map.merge (see context here[1]).
We weren't using the polymorphism anywhere so I figured it was simplest
to just remove it.
- Improve typing around jest mocks throughout the codebase.
Test plan: `yarn test --full` passes.
[1]: https://github.com/flow-typed/flow-typed/issues/2991
This commit integrates an bare skeleton of the odyssey frontend that we
implemented in the [odyssey-hackathon] repository. You can see the
working frontend that we are trying to port over at
[sourcecred.io/odyssey-hackathon/][scio].
The prototype in the other repository has some tooling choices which are
incompatible/redundant with decisions in our codebase (sass vs
aphrodite), and requires some tools not yet present here
(svg-react-loader). This commit includes the build and integration work
needed to port the prototype frontend into mainline SourceCred. The
frontend scaffold isn't yet integrated with any "real" Odyssey data.
One potential issue: right now, every page that is rendered from the
SourceCred homepage is contained within a [homepage/Page], meaning that
it has full SourceCred website styling, along with the SourceCred
website header. The [application][scio] also has a header. Currently, I
work around this by having the Odyssey UI cover up the base header (via
absolute positioning), which works but is hacky. We can consider more
principled solutions:
- Finding a way to specify routes which aren't contained by
[homepage/Page]; maybe by adding a new top-level route
[here][route-alternative].
- Unify the headers for the Odyssey viewer and the page as a whole
(sounds like inappropriate entanglement?)
- Have a website header and also an application header (sounds ugly?)
[homepage/Page]: ee1d2fb996/src/homepage/Page.js
[route-alternative]: ee1d2fb996/src/homepage/createRoutes.js (L17)
Test plan: Run `yarn start`, and then navigate to
`localhost:8080/odyssey/`. observe that a working website is displayed,
and that the cred logo next to the word "SourceCred" is loaded properly
(i.e. svg-react-loader is integrated properly). Observe that there are
no build/compile errors from either `yarn start` or `yarn build`. Also,
observe that the UI looks passably nice, and that if the number of
elements in the entity lists is larger than can be displayed, the
sidebar pane scrolls independently.
The UI was tested in both Chrome and Firefox.
[odyssey-hackathon]: https://github.com/sourcecred/odyssey-hackathon
[scio]: https://sourcecred.io/odyssey-hackathon/
Thanks to @jmnemo, as the implementation is based on [his work].
[his work]: https://github.com/jmnemo/hackathon-event/
This commit adds a module, `fetchGithubOrg`, which loads data on GitHub
organizations, most notably including the list of repositories in that
org.
The structure of this commit is heavily influenced by review feedback
from @wchargin's [review] of a related PR.
Test plan: This logic depends on actually hitting GitHub's API, so the
tests are modeled off the related `fetchGithubRepo` module. There is a
new shell test, `src/plugins/github/fetchGithubOrgTest.sh`, which
verifies that that the org loading logic works via a snapshot.
To verify the correctness of this commit, I've performed the following
checks:
- `yarn test --full` passes
- inspection of `src/plugins/github/example/example-organization.json`
confirms that the list of repositories matches the repos for the
"sourcecred-test-organization" organization
- manually breaking the snapshot (by removing a repo from the snapshot)
causes `yarn test --full` to fail
- running `src/plugins/github/fetchGithubOrgTest.sh -u` restores the
snapshot, afterwhich `yarn test --full` passes again.
[review]: https://github.com/sourcecred/sourcecred/pull/1089#pullrequestreview-204577637
Currently, our underlying test script uses npm rather than yarn to
execute the tests. This is awkward, because we use yarn everywhere else
in lieu of npm. It turns out that some setups have node available
without npm, and in such environments our tests fail with a cryptic
ENOENT error.
This changes the tests to use yarn instead.
Test plan: `yarn test --full` passes.
Thanks to @wpank for help uncovering this issue.
This commit adds a new runOption for execDependencyGraph, namely
`printVerboseResults`. If this flag is true, then execDependencyGraph
will print a "Full Results" section along with the standard error and
standard out of every task, regardless of whether it failed or
succeeded. (Note, this is the existing behavior for all invocations
prior to this commit).
If the flag is not true, then execDependencyGraph will not print a full
results section, and stdout/stderr will be logged only for tasks that
fail.
This commit also modifies `yarn test` to use the new flag so that it
prints verbose tests only when the `--full` option is provided. This is
consistent with our sharness behavior: we print the full sharness logs
only when `--full` was provided.
This fixes#1035, and ensures that running `yarn test` has a high signal
to noise ratio (i.e. it only shows an enumeration of top level tasks).
This improves the developer ergonomics of SourceCred by not having a
super commonly used and core script spam the user with mostly irrelevant
information.
Test plan:
Run `yarn test` when all tests are passing, and observe that the output
has much less noise:
```
yarn run v1.12.3
$ node ./config/test.js
tmpdir for backend output: /tmp/sourcecred-test-6337SZ9smvWsWvqE
Starting tasks
GO ensure-flow-typing
GO check-stopships
GO check-pretty
GO lint
GO flow
GO unit
GO backend
PASS check-stopships
PASS ensure-flow-typing
PASS flow
PASS backend
GO sharness
PASS sharness
PASS check-pretty
PASS lint
PASS unit
Overview
Final result: SUCCESS
Done in 11.66s.
```
Run `yarn test` when there is a real failure (e.g. a unit test failure)
and observe that full details on the failure, including the output from
stdout/stderr, is still provided.
Run `yarn test --full` and observe that full, verbose logs are provided.
As described in #1033: Currently, developers in environments without gnu
coreutils (notably: macOS) get an extremely confusing error message when
they run `yarn test`. This commit ensures that such users instead get a
helpful message with a link to the fix.
Follows @wchargin's proposed approach for fixing #1033, so that Mac
developers get clear guidance on how to get their development
environment working properly.
Expected behavior is that `yarn test` passes when GNU coreutils are
present, and fails with the following message when they are not:
Fixes#1033
```
FAIL check-gnu-coreutils
Exit code: 1
Contents of stdout:
/home/dandelion/git/sc/sourcecred
Contents of stderr:
Error: Your environment does not provide GNU coreutils
You're likely developing on macOS.
Please see the following link for a fix:
https://github.com/sourcecred/sourcecred/issues/698#issuecomment-417202213
```
Test plan:
Verify that on a machine with gnu coreutils present, the command passes
and does not print irrelevant output.
Verify that on a machine without gnu coreutils available, the command
fails, and prints exactly the error message expected, without any extra
output related to the test's implementation.
Paired-with: @anthrocypher
Summary:
Our registry was defined to simply be a list of IDs. This is
insufficiently flexible; we want to be able to annotate these IDs with,
e.g., last-updated times (#989). This commit wraps the entries in a
simple object, updating clients appropriately.
Test Plan:
- Run `node ./bin/sourcecred.js load sourcecred/example-github` with a
repository registry in the old format, and note that it errors
appropriately.
- Run `yarn build` with a repository registry in the old format, and
note that it errors (“Compat mismatch”).
- Delete the old registry and re-run the `load` command. Note that it
runs successfully and outputs a registry. Run `yarn build`; note
that this works.
- Load data for two repositories. Run `yarn start`. Note that the list
of prototypes still works, and that you can navigate to and render
attributions for individual project pages.
- Verify that `yarn test --full` passes.
wchargin-branch: repo-id-registry-metadata
Summary:
There have been some breaking changes that require new type annotations,
which is a good thing: these prevent `any`-leakage.
Test Plan:
Run `yarn flow`.
wchargin-branch: flow-v0.86.0
RepoIdRegistry is used across the project, but not in the explorer. So
it makes very little sense that it live in the explorer module. It's now
moved to core.
Test plan: `yarn test --full` passes
We added a configurable cred feedback url on the theory that we would
create a separate discourse post to collect feedback for each project in
particular.
We've now realized that no one is using this, so it's just vestigial
complexity now. So I'm removing the logic for configuring the feedback
url on a per-project basis.
Instead, we will always link to a Google form for collecting feedback.
Test plan: `yarn test --full` passes, and I manually checked the links.
Summary:
This import does not need to be dynamic; the fact that it is loses us
safety for no benefit. (When I originally wrote it, it was less
obviously bad, but the surrounding code has changed over time.)
Test Plan:
Running `yarn flow` suffices, and now actually checks this module
instead of typing it as `any`. Running `yarn test --full` is nice, too.
wchargin-branch: webpack-remove-dynamic-import
Summary:
We want to remove the repository selector dropdown on the cred explorer
homepage and instead render a separate web page for each project. To do
this, we need to know which pages to render statically. We choose to
ingest this information from the state of the repository registry at
build time.
This commit adds an environment variable `REPO_REGISTRY` whose contents
are the stringified version of the repository registry, or `null` if
SourceCred has been built for the backend. This variable is defined with
Webpack’s `DefinePlugin`, so any code bundled by Webpack can refer to it
via `process.env.REPO_REGISTRY` both on the server and in the browser.
Paired with @wchargin.
Test Plan:
Sharness tests modified; running `yarn test --full` suffices.
Summary:
Prior to this commit, running `yarn test full` (instead of `--full`)
would silently behave the same way as `yarn test`. This is misleading.
It now fails.
Test Plan:
Run `! yarn test full && yarn test --full`.
wchargin-branch: test-validate-arguments
Currently version is located in `homepage/`, which doesn't make much
sense, since it's versioning the whole project.
We move it to core.
Test plan: `yarn test --full`
Summary:
This was used for ad hoc testing of the Mirror module before it was
integrated into SourceCred. We haven’t kept it up to date with schema
changes, and it is no longer needed: you can just run `sourcecred load`.
This was also the only untested code in the `graphql/` package, so it is
nice to remove it.
Test Plan:
Running `yarn test --full` passes.
wchargin-branch: remove-mirror-demo
The referenced pull request mistakenly didn't update `config/paths.js`,
which caused build_static_site to fail in `yarn test full`.
Test plan: `yarn test full` now passes.
Summary:
Fun facts:
- `new Date().getDay()` does not return the day of the month. It
returns the day of the _week_ as an integer `0 ≤ n ≤ 6`.
- `new Date().getDate()` returns the day of the month from 1 to 31.
- `new Date().getMonth()` really does return the month, but _this_ one
is zero-based!
All this to say, my implementation in #901 was a bit flawed.
Why didn’t I notice at the time? I wrote and tested the change on
2018-10-01, which was a Monday, so both `getDay()` and `getDate()` were
in fact `1`. As for me failing to notice that `getMonth()` was off by
one—well, sometimes I’m very dumb.
Test Plan:
```shell
$ NODE_ENV=development node -e '
> require("./config/env");
> console.log(process.env.SOURCECRED_GIT_STATE);
> '
{"commitHash":"f9bb75ef71c5","commitTimestamp":"20181030-1518","dirty":true}
$ date -I
2018-10-30
```
wchargin-branch: env-fix-date-formatting
Summary:
As of this commit, `node ./bin/sourcecred.js load` uses the Mirror code,
and the legacy continuation-fetching code is not included in the
`sourcecred.js` bundle.
We do not yet perform the commit prefetching described in #923. The code
should be plenty fast for repositories that merge pull requests at least
occasionally.
Test Plan:
Running `yarn test --full` passes. Loading `sourcecred/sourcecred` works
and generates a reasonable credit attribution. Loading it again
completes immediately.
wchargin-branch: fetchGithubRepo-mirror
Summary:
This makes significant progress toward #923. As of this commit, it is
possible to use the Mirror module for the whole loading pipeline. This
process may be slow for repositories that do not use pull requests at
all (more precisely, that have large connected commit subgraphs none of
whose nodes is the merge commit of a pull request; see #920 for details)
so it is not yet the default codepath.
Test Plan:
Existing unit tests should suffice. For extra testing, I’ve added a
script that fetches a repository both via the old continuations logic
and the new Mirror logic, then constructs relational views and checks
whether the data is the same. For `example-github`, the views are
identical. For `sourcecred`, they are not: the old continuations logic
erroneously omits two commits, which the Mirror logic includes.
You can run the test like this:
```
$ node ./bin/testContinuations.js \
> sourcecred sourcecred MDEwOlJlcG9zaXRvcnkxMjAxNDU1NzA= \
> /tmp/continuations.json /tmp/mirror.json \
> 2> >(jq . >&2)
{
"child": "0d38dde23a6de831315f3643a7d2bc15e8df7678",
"parent": "cb8ba0eaa1abc1f921e7165bb19e29b40723ce65",
"type": "UNKNOWN_PARENT_OID"
}
{
"child": "d152f48ce4c2ed1d046bf6ed4f139e7e393ea660",
"parent": "de7a8723963d9cd0437ef34f5942a071b850c0e7",
"type": "UNKNOWN_PARENT_OID"
}
Different. Saving to disk...
```
Use `diff -u <(jq . /tmp/continuations.json) <(jq . /tmp/mirror.json)`
to inspect the differences, and note that exactly the two missing
commits have been added and that there are no other changes. (The diff
is small: just 51 lines of nicely formatted JSON.) The full log is here:
<https://gist.github.com/wchargin/e159cac9dcf3cc3b1efbd54f59e24e0b>
I also generated the `sourcecred/sourcecred` cred attribution and viewed
it with `yarn start`, which seems to work fine.
wchargin-branch: relationalview-new-data-format
Summary:
The included schema is forked from the one in `graphql/demo.js`.
Primitive types have been added, and the `parents` connection has been
added to commit objects per #920. (We do not include this in the demo
script because without prefetching it would take a long time to load.)
Test Plan:
Unit tests added; run `yarn unit`. Then run `yarn backend` and verify
that `node ./bin/generateGithubGraphqlFlowTypes.js` generates exactly
the same output as in the types file:
```
$ node ./bin/generateGithubGraphqlFlowTypes.js |
> diff -u - ./src/plugins/github/graphqlTypes.js
$ echo $?
0
```
Change the `graphqlTypes.js` file and verify that `yarn unit` fails.
As the build config has been changed, a `yarn test --full` is warranted.
It passes.
Finally, I have manually verified that the schema is consistent with the
documentation at <https://developer.github.com/v4/object/repository/>
and related pages.
wchargin-branch: github-schema-flow-types
Summary:
Closes#902. CircleCI continues to work fast and well, on both the
commit and nightly workflows. Travis continues to be slower. It is time.
I have disabled Travis for `sourcecred/sourcecred` via:
<https://travis-ci.org/profile/sourcecred>
This commit removes the vestigial configuration files and code.
Test Plan:
Running `yarn test` still works. Running `yarn test --full` still works,
and properly invokes `sharness-full` and the various extra tests.
Running `git grep -i travis` yields no results.
wchargin-branch: ci-remove-travis
Summary:
This completes the minimum viable public API for the `Mirror` class. As
described on the docstring, the typical workflow for a client is:
- Invoke the constructor to acquire a `Mirror` instance.
- Invoke `registerObject` to register a root object of interest.
- Invoke `update` to update all transitive dependencies.
- Invoke `extract` to retrieve the data in structured form.
It is the third step that is added in this commit.
In this commit, we also include a command-line demo of the Mirror
module, which exercises exactly the workflow above with a hard-coded
GitHub schema. This can be used to test the module’s behavior with
real-world data. I plan to remove this entry point once we integrate
this module into SourceCred.
This commit makes progress toward #622.
Test Plan:
Unit tests included for the core functionality. The imperative shell
does not have automated tests. You can test it as follows.
First, run `yarn backend` to build `bin/mirror.js`. Then, run:
```shell
$ node bin/mirror.js /tmp/mirror-demo.db \
> Repository MDEwOlJlcG9zaXRvcnkxMjMyNTUwMDY= \
> 60
```
Here, the big base64 string is the ID for the sourcecred/example-github
repository. (This is listed in `graphql/demo.js`, alongside the ID for
the SourceCred repository itself.) The value 60 is a TTL in seconds. The
database filename is arbitrary.
This will print out a ton of output to stderr (all intermediate queries
and responses, for debugging purposes), and then the full contents of
the example repository to stdout.
Run the command again, and it should finish instantly. On my machine,
the main function runs faster than the Node startup time (50ms vs 60ms).
Then, re-run the command, changing the TTL from `60` to `1`. Note that
it sends off some queries and then prints the same repository.
It is safe to kill the program at any time, either while waiting for a
response from GitHub or while processing the results, because the mirror
module takes advantage of SQLite’s transaction-safety. Intermediate
updates will be persisted, so you’ll lose just a few seconds of
progress.
You can also of course dive into the generated database yourself to
explore the data. It’s good fun.
wchargin-branch: mirror-e2e-update
Summary:
Git only learned `--date=format:...` in Git 2.6.0. Some old Docker
images have older versions of Git. It’s not too much work to reimplement
this particular bit of functionality, so this commit does so.
Test Plan:
Install Git 2.1.4 (as used on CircleCI): from the Git repository for
Git itself, run:
git checkout v2.1.4
make
make install
Watch `yarn test --full` pass. Before this commit, `yarn unit` failed.
wchargin-branch: env-support-git-2.1.4
Summary:
Our environment-stripping logic used in `config/env.js` to read the
current Git state included stripping the `PATH` environment variable.
This had the effect that the system Git executable would always be used
in preference to a user-installed version.
Test Plan:
Run `/usr/bin/git --version`, and then install a different version of
Git. (For instance, check out an old tag, then `make && make install`
from the `git/git` repository.) Then, add
```js
console.log(execFileSync("git", ["--version"], {env}).toString());
```
to `getGitState` in `config/env.js`, and run
NODE_ENV=development node ./config/env.js
Note that this prints the version of the system Git before this change,
and the user Git after this change.
Alternately, local-install a version of Git earlier than 2.6.0, and note
that `yarn unit` now _fails_ because the `--date=format:…` syntax is not
known to such versions of Git. Prior to this commit, the tests would
pass as long as the system Git were more recent.
wchargin-branch: env-git-path
This modifies the behavior when loading the Git plugin so that it
serializes the Repository as well as the graph. This will allow us to
get extra information, like the commit headline, to the Git plugin in
the frontend.
As an added bonus, we can now refactor `loadRepositoryTest` to depend on
`sourcecred.js load` rather than `loadAndPrintRepository`. As this was
the only use for `loadAndPrintRepository`, we can safely delete it. This
improves our test quality because it means we are also testing the
actual CLI behavior.
Note that the switch from using `stringify` to `json.tool` for
pretty-printing has resulted in a trivial diff in the snapshot.
Test plan: `yarn test --full` passes.
According to the [jest docs], setting `maxWorkers=4` can substantially
improve performance on CI.
This commit sets `maxWorkers=4` when running `yarn unit` as a part of
`yarn test`. Based on my local testing (see data below), this improves
performance locally in addition to the expected performance improvement
on travis.
\## Testing `yarn unit` by itself
```
yarn unit --maxWorkers=1
15.28s user 3.71s system 112% cpu 16.848 total
15.13s user 3.68s system 112% cpu 16.708 total
15.32s user 3.76s system 112% cpu 16.917 total
15.91s user 3.74s system 112% cpu 17.425 total
15.61s user 3.76s system 113% cpu 17.125 total
yarn unit --maxWorkers=2
19.43s user 4.03s system 212% cpu 11.061 total
19.86s user 4.19s system 210% cpu 11.407 total
21.19s user 4.26s system 213% cpu 11.902 total
20.68s user 4.51s system 212% cpu 11.873 total
20.78s user 4.26s system 212% cpu 11.780 total
yarn unit --maxWorkers=4
29.43s user 5.14s system 389% cpu 8.865 total
29.99s user 5.18s system 392% cpu 8.961 total
32.22s user 5.52s system 390% cpu 9.659 total
33.99s user 5.97s system 395% cpu 10.097 total
33.38s user 5.93s system 395% cpu 9.933 total
yarn unit --maxWorkers=8
48.21s user 6.57s system 621% cpu 8.815 total
51.61s user 7.16s system 610% cpu 9.622 total
59.48s user 7.82s system 621% cpu 10.833 total
58.18s user 8.10s system 624% cpu 10.607 total
58.92s user 8.22s system 620% cpu 10.817 total
unset
46.27s user 6.44s system 599% cpu 8.799 total
49.08s user 7.04s system 600% cpu 9.342 total
54.85s user 7.52s system 600% cpu 10.383 total
55.66s user 7.52s system 605% cpu 10.438 total
53.77s user 7.50s system 604% cpu 10.142 total
```
\## Testing `yarn test`
```
maxWorkers=1
46.65s user 5.92s system 249% cpu 21.038 total
47.94s user 5.81s system 251% cpu 21.354 total
51.50s user 6.44s system 260% cpu 22.234 total
52.60s user 6.65s system 268% cpu 22.077 total
53.04s user 6.27s system 266% cpu 22.278 total
maxWorkers=2
56.13s user 6.13s system 409% cpu 15.204 total
63.32s user 7.22s system 412% cpu 17.091 total
64.82s user 7.19s system 422% cpu 17.027 total
64.59s user 7.41s system 417% cpu 17.227 total
65.40s user 7.30s system 419% cpu 17.318 total
maxWorkers=4
74.64s user 7.60s system 584% cpu 14.066 total
82.69s user 8.43s system 582% cpu 15.643 total
85.00s user 8.68s system 591% cpu 15.835 total
84.81s user 8.58s system 595% cpu 15.690 total
85.22s user 8.59s system 596% cpu 15.719 total
maxWorkers=4 and everything depends on unit
59.29s user 6.01s system 378% cpu 17.261 total
62.99s user 6.64s system 375% cpu 18.564 total
65.54s user 7.31s system 375% cpu 19.419 total
63.24s user 7.13s system 379% cpu 18.548 total
63.68s user 7.13s system 383% cpu 18.457 total
maxWorkers=8
92.85s user 8.13s system 643% cpu 15.702 total
101.63s user 9.21s system 632% cpu 17.510 total
101.63s user 9.23s system 636% cpu 17.428 total
101.81s user 9.32s system 633% cpu 17.546 total
101.62s user 9.39s system 632% cpu 17.542 total
unset
88.75s user 8.15s system 646% cpu 14.988 total
96.43s user 9.23s system 631% cpu 16.739 total
98.27s user 9.17s system 638% cpu 16.819 total
98.46s user 9.01s system 642% cpu 16.729 total
98.53s user 9.15s system 637% cpu 16.889 total
unset + everything depends on unit
76.02s user 7.61s system 486% cpu 17.208 total
79.14s user 8.26s system 484% cpu 18.030 total
84.32s user 9.19s system 488% cpu 19.136 total
84.92s user 9.14s system 497% cpu 18.919 total
84.46s user 8.94s system 492% cpu 18.965 total
```
Test plan: `yarn test` passes here and on travis
[jest docs]: https://jestjs.io/docs/en/troubleshooting
Summary:
CI mode prevents Jest from automatically writing snapshots, and also
causes obsolete snapshots to be an error instead of a warning. This is
consistent with the behavior on Travis, where the `CI=1` environment
variable is set. It should thus be the default when running `yarn test`
(but not `yarn unit`).
Test Plan:
Add a file `src/foo.test.js`:
```js
// @flow
describe("foo", () => {
it("bar", () => {
expect("baz").toMatchSnapshot();
});
});
```
Note that `yarn test` fails with the message, “new snapshot was not
written”.
Revert this change, then re-run `yarn test`; note that it passes,
writing a snapshot.
Then, reinstate this change and delete `src/foo.test.js`. Note that
running `yarn test` fails, due to an obsolete snapshot. Revert the
change again, and watch `yarn test` pass despite the obsolete snapshot.
Finally, remove the snapshot. :-)
wchargin-branch: test-jest-ci
Dandelion Mané
William Chargin
