Commit Graph

50 Commits

Author SHA1 Message Date
Dandelion Mané 669f34d009
Add `fetchGithubOrg` for loading organizations (#1117)
This commit adds a module, `fetchGithubOrg`, which loads data on GitHub
organizations, most notably including the list of repositories in that
org.

The structure of this commit is heavily influenced by review feedback
from @wchargin's [review] of a related PR.

Test plan: This logic depends on actually hitting GitHub's API, so the
tests are modeled off the related `fetchGithubRepo` module. There is a
new shell test, `src/plugins/github/fetchGithubOrgTest.sh`, which
verifies that that the org loading logic works via a snapshot.

To verify the correctness of this commit, I've performed the following
checks:

- `yarn test --full` passes
- inspection of `src/plugins/github/example/example-organization.json`
confirms that the list of repositories matches the repos for the
"sourcecred-test-organization" organization
- manually breaking the snapshot (by removing a repo from the snapshot)
causes `yarn test --full` to fail
- running `src/plugins/github/fetchGithubOrgTest.sh -u` restores the
snapshot, afterwhich `yarn test --full` passes again.

[review]: https://github.com/sourcecred/sourcecred/pull/1089#pullrequestreview-204577637
2019-03-19 19:00:08 -07:00
William Chargin 415210b772
webpack: remove dynamic import (#982)
Summary:
This import does not need to be dynamic; the fact that it is loses us
safety for no benefit. (When I originally wrote it, it was less
obviously bad, but the surrounding code has changed over time.)

Test Plan:
Running `yarn flow` suffices, and now actually checks this module
instead of typing it as `any`. Running `yarn test --full` is nice, too.

wchargin-branch: webpack-remove-dynamic-import
2018-11-01 13:24:31 -07:00
William Chargin 64500f53cb
mirror: remove "demo" module (#978)
Summary:
This was used for ad hoc testing of the Mirror module before it was
integrated into SourceCred. We haven’t kept it up to date with schema
changes, and it is no longer needed: you can just run `sourcecred load`.

This was also the only untested code in the `graphql/` package, so it is
nice to remove it.

Test Plan:
Running `yarn test --full` passes.

wchargin-branch: remove-mirror-demo
2018-11-01 11:29:57 -07:00
Dandelion Mané 1beec07e40
Fix a build failure induced by #974 (#975)
The referenced pull request mistakenly didn't update `config/paths.js`,
which caused build_static_site to fail in `yarn test full`.

Test plan: `yarn test full` now passes.
2018-11-01 10:54:53 -07:00
William Chargin 08219f98bf
fetchGithubRepo: use Mirror pipeline (#937)
Summary:
As of this commit, `node ./bin/sourcecred.js load` uses the Mirror code,
and the legacy continuation-fetching code is not included in the
`sourcecred.js` bundle.

We do not yet perform the commit prefetching described in #923. The code
should be plenty fast for repositories that merge pull requests at least
occasionally.

Test Plan:
Running `yarn test --full` passes. Loading `sourcecred/sourcecred` works
and generates a reasonable credit attribution. Loading it again
completes immediately.

wchargin-branch: fetchGithubRepo-mirror
2018-10-28 12:03:06 -07:00
William Chargin e2c99c418b
relationalView: use new data format (#934)
Summary:
This makes significant progress toward #923. As of this commit, it is
possible to use the Mirror module for the whole loading pipeline. This
process may be slow for repositories that do not use pull requests at
all (more precisely, that have large connected commit subgraphs none of
whose nodes is the merge commit of a pull request; see #920 for details)
so it is not yet the default codepath.

Test Plan:
Existing unit tests should suffice. For extra testing, I’ve added a
script that fetches a repository both via the old continuations logic
and the new Mirror logic, then constructs relational views and checks
whether the data is the same. For `example-github`, the views are
identical. For `sourcecred`, they are not: the old continuations logic
erroneously omits two commits, which the Mirror logic includes.

You can run the test like this:

```
$ node ./bin/testContinuations.js \
> sourcecred sourcecred MDEwOlJlcG9zaXRvcnkxMjAxNDU1NzA= \
> /tmp/continuations.json /tmp/mirror.json \
> 2> >(jq . >&2)
{
  "child": "0d38dde23a6de831315f3643a7d2bc15e8df7678",
  "parent": "cb8ba0eaa1abc1f921e7165bb19e29b40723ce65",
  "type": "UNKNOWN_PARENT_OID"
}
{
  "child": "d152f48ce4c2ed1d046bf6ed4f139e7e393ea660",
  "parent": "de7a8723963d9cd0437ef34f5942a071b850c0e7",
  "type": "UNKNOWN_PARENT_OID"
}
Different. Saving to disk...
```

Use `diff -u <(jq . /tmp/continuations.json) <(jq . /tmp/mirror.json)`
to inspect the differences, and note that exactly the two missing
commits have been added and that there are no other changes. (The diff
is small: just 51 lines of nicely formatted JSON.) The full log is here:
<https://gist.github.com/wchargin/e159cac9dcf3cc3b1efbd54f59e24e0b>

I also generated the `sourcecred/sourcecred` cred attribution and viewed
it with `yarn start`, which seems to work fine.

wchargin-branch: relationalview-new-data-format
2018-10-23 16:42:49 -07:00
William Chargin 889febb7f6
github: add GraphQL schema and Flow types (#928)
Summary:
The included schema is forked from the one in `graphql/demo.js`.
Primitive types have been added, and the `parents` connection has been
added to commit objects per #920. (We do not include this in the demo
script because without prefetching it would take a long time to load.)

Test Plan:
Unit tests added; run `yarn unit`. Then run `yarn backend` and verify
that `node ./bin/generateGithubGraphqlFlowTypes.js` generates exactly
the same output as in the types file:

```
$ node ./bin/generateGithubGraphqlFlowTypes.js |
> diff -u - ./src/plugins/github/graphqlTypes.js
$ echo $?
0
```

Change the `graphqlTypes.js` file and verify that `yarn unit` fails.

As the build config has been changed, a `yarn test --full` is warranted.
It passes.

Finally, I have manually verified that the schema is consistent with the
documentation at <https://developer.github.com/v4/object/repository/>
and related pages.

wchargin-branch: github-schema-flow-types
2018-10-19 09:04:54 -07:00
William Chargin 55950f5354
mirror: add an end-to-end `update` function (#909)
Summary:
This completes the minimum viable public API for the `Mirror` class. As
described on the docstring, the typical workflow for a client is:

  - Invoke the constructor to acquire a `Mirror` instance.
  - Invoke `registerObject` to register a root object of interest.
  - Invoke `update` to update all transitive dependencies.
  - Invoke `extract` to retrieve the data in structured form.

It is the third step that is added in this commit.

In this commit, we also include a command-line demo of the Mirror
module, which exercises exactly the workflow above with a hard-coded
GitHub schema. This can be used to test the module’s behavior with
real-world data. I plan to remove this entry point once we integrate
this module into SourceCred.

This commit makes progress toward #622.

Test Plan:
Unit tests included for the core functionality. The imperative shell
does not have automated tests. You can test it as follows.

First, run `yarn backend` to build `bin/mirror.js`. Then, run:

```shell
$ node bin/mirror.js /tmp/mirror-demo.db \
> Repository MDEwOlJlcG9zaXRvcnkxMjMyNTUwMDY= \
> 60
```

Here, the big base64 string is the ID for the sourcecred/example-github
repository. (This is listed in `graphql/demo.js`, alongside the ID for
the SourceCred repository itself.) The value 60 is a TTL in seconds. The
database filename is arbitrary.

This will print out a ton of output to stderr (all intermediate queries
and responses, for debugging purposes), and then the full contents of
the example repository to stdout.

Run the command again, and it should finish instantly. On my machine,
the main function runs faster than the Node startup time (50ms vs 60ms).

Then, re-run the command, changing the TTL from `60` to `1`. Note that
it sends off some queries and then prints the same repository.

It is safe to kill the program at any time, either while waiting for a
response from GitHub or while processing the results, because the mirror
module takes advantage of SQLite’s transaction-safety. Intermediate
updates will be persisted, so you’ll lose just a few seconds of
progress.

You can also of course dive into the generated database yourself to
explore the data. It’s good fun.

wchargin-branch: mirror-e2e-update
2018-10-02 21:06:01 -07:00
Dandelion Mané e3f04c5079
Git plugin serializes the repository and graph (#874)
This modifies the behavior when loading the Git plugin so that it
serializes the Repository as well as the graph. This will allow us to
get extra information, like the commit headline, to the Git plugin in
the frontend.

As an added bonus, we can now refactor `loadRepositoryTest` to depend on
`sourcecred.js load` rather than `loadAndPrintRepository`. As this was
the only use for `loadAndPrintRepository`, we can safely delete it. This
improves our test quality because it means we are also testing the
actual CLI behavior.

Note that the switch from using `stringify` to `json.tool` for
pretty-printing has resulted in a trivial diff in the snapshot.

Test plan: `yarn test --full` passes.
2018-09-20 14:07:58 -07:00
William Chargin e71264f5cc
Replace `oclif` with `cli` (#744)
Summary:
This commit changes the CLI to use the code in `cli` instead of `oclif`.
A subsequent commit will remove the dependency on OClif altogether.
Resolves #580.

Test Plan:
Note that `yarn backend; node bin/sourcecred.js help` works. Note that
the documentation in the README is still correct.

wchargin-branch: cli-replace-oclif
2018-09-02 16:11:56 -07:00
William Chargin ff2d4f2fd8
cli: add `main`, `sourcecred`, and `help` (#741)
Summary:
This commit includes a minimal usage of an actual CLI application. It
provides the `help` command and no actual functionality.

Test Plan:
Unit tests added, with full coverage. To see it in action, first run
`yarn backend`, then run `node bin/cli.js help`.

wchargin-branch: cli-beginnings
2018-09-02 15:53:24 -07:00
William Chargin 1f4a6395c8
cli: rename existing system from `cli` to `oclif` (#739)
Summary:
Per #580, we aim to remove OClif. To do so, we move the old system to a
directory `oclif`, and will create the new system in the now-vacant
`cli` directory.

Test Plan:
Note that `yarn backend` still builds, that `node bin/sourcecred.js`
still has `help` and `load`, and that `git grep -wc cli` yields only
`yarn.lock:9`.

wchargin-branch: rename-cli-to-oclif
2018-09-02 15:44:30 -07:00
William Chargin c84a1c01e8
Remove remaining public URL logic (#686)
Summary:
Now that the main functionality of #643 has been implemented, we no
longer have any use for the “public URL” property. In fact, its presence
is actively harmful, as it suggests that the gateway may be known before
runtime, which is confusing and false.

Closes #643.

Test Plan:
Running `yarn start` works. Building the static site works.
Invoking `git grep -i 'public.\?url'` finds no matches.
Also, `yarn test --full` passes.

wchargin-branch: remove-public-url
2018-08-16 11:19:09 -07:00
William Chargin 3eb2b6eec6
Add a favicon (#637)
Summary:
In addition to the obvious benefit of having a favicon, this gets rid of
a 404 Not Found error on our home page, tremendously boosting our hacker
cred.

Test Plan:
The favicon is displayed in both `yarn start` and the static site (as a
result of the build script). The added build test fails before this
change.

wchargin-branch: add-favicon
2018-08-10 13:15:49 -07:00
William Chargin 7a4401e3ef
Remove the `start` CLI command and dependencies (#613)
Summary:
We never use the `node ./bin/sourcecred.js start` command. This command
contains an Express server to combine the static files with the build
output, which duplicates the logic in our Webpack config, which we
actually use (with `yarn start`). Once we actually want the command line
entry point to be a useful tool for end users, we can consider
reimplementing it the right way, whatever that may be. Until then, it’s
simply one more thing to keep in sync.

Test Plan:
Running `yarn test --full` passes; the `load` CLI command still works;
running `yarn start` still works.

wchargin-branch: remove-start
2018-08-07 11:00:09 -07:00
William Chargin 9d5c4454c5
Remove `src/app/public` (#582)
Summary:
This subtree has no effect on the new build process; it contains only
stale code.

Test Plan:
Running `yarn test --full` passes. Running `yarn build` and running an
HTTP server on the result indicates the expected behavior, as does
running `yarn start`. A quick `git grep public` finds no amok results.

wchargin-branch: remove-public
2018-07-31 21:45:25 -07:00
William Chargin b45ef739fe
new-webpack: clean `build/` before prod (#568)
Summary:
In our current system, we build by invoking `scripts/build.js`, which
begins by removing the `build/` directory. This behavior is nice,
because it prevents cross-contamination between builds. In this commit,
we add a plugin to achieve the same result from directly within Webpack.

Test Plan:
Run

```
mkdir -p ./build
touch ./build/wat
NODE_ENV=production node ./node_modules/.bin/webpack \
    --config config/makeWebpackConfig.js
```

and ensure that `./build/wat` does not exist after the build completes.

wchargin-branch: webpack-clean-build
2018-07-30 18:01:47 -07:00
William Chargin 9a356f88a1 Use route data for static site source of truth
Summary:
This removes the hard-coded route data from the Webpack config,
replacing it with the list of paths exported by the route data module.

Test Plan:
Note that the output of `yarn build` is identical before and after this
change: namely,

```shell
$ find build -exec shasum -a 256 {} + | shasum -a 256
7610a61f8a977f1d8edd849fc81256ca15f41f366e5fdb4b59a5d5ce37d6d58e
```

wchargin-branch: non-hard-coded-route-data
2018-07-23 13:29:48 -07:00
William Chargin b41009b1f7 Implement a first-pass static site generation
Summary:
Some of the code here is adapted from my site (source available on
GitHub at wchargin/wchargin.github.io). It has been improved when
possible and made worse when necessary to fit into our existing build
system with minimal churn.

As of this commit, there remain the following outstanding tasks:
  - Use a non-hardcoded list of paths in static site generation router.
    This is not trivial. We have the paths nicely available in
    `routes.js`, but this module is written in ES6, and transitively
    depends on many files written in ES6 (i.e., the whole app). Yet
    naïvely it would be required from a Webpack config file, which is
    interpreted as vanilla JavaScript.
  - Add `csso-loader` to minify our CSS. This is easy.
  - Add unit tests for `dedent`. (As is, it comes from my site
    verbatim. I wrote it. dmnd’s `dedent` package on npm is insufficient
    because it dedents arguments as well as the format string, which is
    incorrect at least for our purposes.)
  - Link in canonical static data for the site.
  - Rip out the whole build system and replace it with my build config,
    which is orders of magnitude saner and less bad. (By “the whole
    build system” I mostly mean `webpack.config.{dev,prod}.js`.)

Test Plan:

```shell
$ yarn backend
$ yarn build
$ node ./bin/sourcecred.js start
```

wchargin-branch: static-v0
2018-07-23 13:29:48 -07:00
Dandelion Mané 664b8ac8d0
Remove testsSetup from config/paths (#475)
It references a non-existent file.

Test plan: `yarn travis --full` passed.
2018-07-01 22:13:10 -07:00
Dandelion Mané 24cf35da22
Change `src/v3/` to `src/` and remove v3 naming (#474)
Test plan:
`git grep -i v3` only shows incidental hits in longer strings
`yarn travis --full` passes
`yarn backend` works
`yarn build` works
`yarn start` works
`node bin/sourcecred.js start` works
`node bin/sourcecred.js load sourcecred example-github` works

Paired with @wchargin
2018-06-30 16:01:54 -07:00
William Chargin 23704da7a5
Demolish the bridge (#473)
Summary:
The bridge introduced in #448 has now served its purpose, and may be
deconstructed. This implements the first part of the last step of the
plan described in that pull request.

Paired with @decentralion.

Test Plan:
After `yarn backend && yarn build`:
  - `node bin/sourcecred.js start` works, and
  - `yarn start` works, and
  - `yarn travis --full` works.

wchargin-branch: demolish-bridge
2018-06-30 15:56:36 -07:00
Dandelion Mané efefc73e6b
Delete the `v1` and `v2` directories from #327 (#472)
Test plan:
`node bin/sourcecred.js load sourcecred example-github` works
`yarn start` works
`node bin/sourcecred.js start-v3` works
`yarn travis --full` passes

Paired with @wchargin
2018-06-30 15:38:47 -07:00
William Chargin 0300a805fa
Copy `start` to `start-v3` (#471)
Summary:
This could also be moved into the bridge directory, but this way is
marginally easier, and it doesn’t really matter in the end.

Test Plan:
`yarn backend` followed by `node bin/sourcecredV3.js start-v3` works.

wchargin-branch: start-v3
2018-06-30 15:28:13 -07:00
Dandelion Mané addaf4e2a8
Add load CLI command (#470)
The `load` command replaces `plugin-load`. By default, it loads data for
all plugins, and does so in parallel using execDependencyGraph. If
passed the optional `--plugin` flag, then it will load data just for
that plugin.

As an implementation detail, when loading all plugins, load calls itself
with the plugin flag set.

Usage:
`node bin/sourcecred.js load repoOwner repoName`

Test plan:
Tested by hand; I blew away my SourceCred directory and then loaded the
example-github repository.
2018-06-30 15:16:35 -07:00
William Chargin bb75cc54cd
Move Express server API from V1 to bridge (#469)
Test Plan:
`yarn start` and `node bin/sourcecred.js start` both still work.

wchargin-branch: bridge-api
2018-06-30 15:11:52 -07:00
William Chargin ca5346b524
Create a bridge for the V1 and V3 apps (#448)
Summary:
Our build system doesn’t make it easy to have two separate React
applications, which we would like to have for the V1 and V3 branches.
Instead, we’ll implement a bridge to maintain compatibility.

The plan looks like this:

 1. Change the app from pointing to V1 to pointing to a bridge
 2. Move the router into the bridge and move the V1 app from the `/`
    route to the `/v1` route (e.g., `/v1/explorer`)
 3. Add a V3 app under the `/v3` route
 4. ???
 5. Delete the V1 app and remove it from the bridge
 6. Delete the bridge and move the V3 app from the `/v3` route to `/`

This commit implements Step 1.

Test Plan:
To verify that the bridge is in fact showing, apply

```diff
diff --git a/src/bridge/app/index.js b/src/bridge/app/index.js
index 379e289..72e784c 100644
--- a/src/bridge/app/index.js
+++ b/src/bridge/app/index.js
@@ -9,5 +9,11 @@ const root = document.getElementById("root");
 if (root == null) {
   throw new Error("Unable to find root element!");
 }
-ReactDOM.render(<V1App />, root);
+ReactDOM.render(
+  <React.Fragment>
+    <h1>Hello</h1>
+    <V1App />
+  </React.Fragment>,
+  root
+);
 registerServiceWorker();
```

and say “hello” back to the app.

wchargin-branch: bridge
2018-06-29 13:09:39 -07:00
William Chargin 4184e8594a
Save the GitHub relational store from the CLI (#447)
Summary:
This provides a command-line entry point `load-plugin-v3` (which will
become `load-plugin` eventually), which fetches the GitHub data via
GraphQL and saves the resulting `RelationalStore` to disk.

A change to the Babel config is needed to prevent runtime errors of the
form `_callee7` is not defined, where `_callee7` is a gensym that is
appears exactly once in the source (in use position, not definition
position). I’m not sure exactly what is causing the error or why this
config change fixes it. But while this patch may be fragile, I don’t
think that it’s likely to subtly break anything, so I’m okay with
pushing it for now and dealing with any resulting breakage as it arises.

Paired with @decentralion.

Test Plan:
Run `yarn backend`, then run something like:

```
node bin/sourcecredV3.js load-plugin-v3 \
    sourcecred example-github --plugin github
```

Inspect results in `SOURCECRED_DIR/data/OWNER/NAME/github/view.json`,
where `SOURCECRED_DIR` is `/tmp/sourcecred` by default, and `OWNER` and
`NAME` are the repository owner and name.

This example repository takes about 1.1 seconds to run. The SourceCred
repository takes about 45 seconds.

wchargin-branch: cli-load-plugin
2018-06-29 12:12:37 -07:00
William Chargin 3835862f82
Create a V3 command-line entry point (#446)
Summary:
Due to oclif’s structure, this entry point shares its `commands`
directory with that of the V1 entry point. We’ll therefore add commands
like `start-v3` as we go.

Test Plan:
`yarn backend` works, and `node bin/sourcecredV3.js start` launches the
V1 server.

wchargin-branch: v3-cli
2018-06-29 11:47:24 -07:00
William Chargin 9347348dd7
Copy graph-independent V1 Git plugin code to V3 (#401)
Summary:
Many files are unchanged. Some files have had paths updated, or new
build/test targets added.

The `types.js` file includes payload type definitions. These are
technically independent of the graph abstraction (i.e., nothing from V1
is imported and the code all still works), but it of course implicitly
depends on the V1 model. For now, we include the entirety of this file,
just so that we have a clean copy operation. Subsequent commits will
strip out this extraneous code.

Suggest reviewing with the `--find-copies-harder` argument to Git’s
diffing functions.

Test Plan:
Running `yarn travis --full` passes. Running

    ./src/v3/plugins/git/demoData/synchronizeToGithub.sh --dry-run

yields “Everything up-to-date”.

wchargin-branch: git-v3-copy
2018-06-20 15:28:37 -07:00
Dandelion Mané 0339d9f41b
Port GitHub data ingestion into v3 (#378)
This commit copies the following logic necessary for downloading GitHub
data into v3. Minimal changes have been made to accomodate the new path
structure.

Test plan:
- Manually ran plugins/github/fetchGithubRepoTest.sh and verified that
it can correctly pass and fail
- Added the v3 github repo test to `yarn travis --full`
- Ran `yarn travis --full` and it passed

Paired with @wchargin
2018-06-11 18:57:37 -07:00
Dandelion Mané ba721a6fbb
Fork project to v1/ and v2/ in preparation for v3 (#327)
We want to reset some of our basic assumptions, and make `Graph` into a
pure graph implementation, rather than a hybrid graph and key-value
store.

This is a substantial rewrite, so we want to start from scratch in a v3/
directory and pull code into v3 as necessary. So that we can do this in
a relatively clean fashion, we're first moving the v1 and v2 code into
their own directories.

Paired with @wchargin

Test plan:  Travis, and `yarn backend`, `node bin/sourcecred.js start`.

Note that `yarn backend` and `node bin/sourcecred.js start` both use the
v1 versions. We'll migrate those (by changing paths.js) to v3 when
appropriate.
2018-06-01 17:17:44 -07:00
William Chargin 9ea1f981aa
Proxy Webpack dev server through to an API server (#245)
Summary:
This way, our frontend can talk to a backend that can read from the
filesystem (among other things).

Paired with @decentralion.

Test Plan:
```
$ yarn backend
$ SOURCECRED_DIRECTORY=/tmp/srccrd yarn start
$ # verify that the browser looks good
$ mkdir /tmp/srccrd
$ echo hello >/tmp/srccrd/world
$ curl localhost:3000/api/v1/data/world
hello
$ curl localhost:4000/api/v1/data/world
hello
```

wchargin-branch: webpack-proxy
2018-05-08 16:09:37 -07:00
Dandelion Mané d221a933d8
Fix flow errors in paths.js (#238)
- Fix accidental string-to-NaN coercion in ensureSlash
- Don't dynamically require package.json; to determine public url, just
use the environment variable or "/"

Test plan: `yarn start` and travis still work
2018-05-08 14:21:19 -07:00
Dandelion Mané 63351e6149 Move app scaffolding to src/app
This commit executes a micro-refactor to move all top-level app setup
code out of src/plugins/artifact/editor and into src/app. The observed
behavior from `yarn start`, which is to show the artifact editor, is
unchanged.
2018-05-08 12:55:38 -07:00
William Chargin 57682065fd
Add `sourcecred start` (#234)
Summary:
We need a way for our web applications to interact with data on the
filesystem. In this commit, we introduce a webserver that serves
statically from two directory trees: first, the result of a live-updated
Webpack build; second, the SourceCred data directory.

Test Plan:
Run `yarn backend` and `node ./bin/sourcecred.js start`. When ready,
navigate to the server’s root route in a web browser. Note that a nice
React app is displayed. Then, change something in that React app source.
Note that the server console displays Webpack’s update messages, and
that refreshing the page in the browser renders the new version of the
app. Finally, visit

    /__data__/graphs/sourcecred/example-github/graph.json

in the browser to see the graph for the example repository, assuming
that you had generated its graph previously.

wchargin-branch: start
2018-05-07 20:10:49 -07:00
Dandelion Mané 93e2798f37
Ensure that flow is used in all js files (#232)
This script ensures that either //@flow or //@no-flow is present in
every js file. Every existing js file that would fail this check has
been given //@no-flow, we should work to remove all of these in the
future.

Test plan:
I verified that `yarn travis` fails before fixing the other js files,
and passes afterwards.
2018-05-07 20:02:19 -07:00
Dandelion Mané 61635a14a7
Remove redundant scripts (#225)
Our SourceCred CLI tool now ipmlements printCombinedGraph and
cloneAndPrintGitGraph, but with more principled implementations and
interfaces :)

Test plan:
`yarn travis --full` passes, so I didn't delete any needed test infra.
2018-05-07 16:15:00 -07:00
William Chargin 2aeeca9a13
Implement a command-line interface (#217)
Summary:
This commit implements the `sourcecred` command-line utility, which has
three subcommands:
  - `plugin-graph` creates one plugin’s graph;
  - `combine` combines multiple on-disk graphs; and
  - `graph` creates all plugins’ graphs and combines them.

As an implementation detail, the `into.sh` script is very convenient,
avoiding needing to do any pipe management in Node (which is Not Fun).
When we build for release, we may want to factor that differently.

Test Plan:
To see it all in action, run `yarn backend`, and then try:
```
$ export SOURCECRED_GITHUB_TOKEN="your_token_here"
$ node ./bin/sourcecred.js graph sourcecred sourcecred
Using output directory: /tmp/sourcecred/sourcecred

Starting tasks
  GO   create-git
  GO   create-github
 PASS  create-github
 PASS  create-git
  GO   combine
 PASS  combine

Full results
 PASS  create-git
 PASS  create-github
 PASS  combine

Overview
Final result:  SUCCESS

$ ls /tmp/sourcecred/sourcecred/
graph-github.json  graph-git.json  graph.json

$ jq '.nodes | length' /tmp/sourcecred/sourcecred/*.json
1000
7302
8302
```
The `node sourcecred.js graph` command takes 9.8s for me.

(The salient point of the last command is that the two small graphs have
node count adding up to the node count of the big graph. Incidentally,
we are [almost][1] at a nice round number of nodes in the GitHub graph.)

[1]: https://xkcd.com/1000/

wchargin-branch: cli
2018-05-07 12:23:09 -07:00
Dandelion Mané fa4082c95b
Minimal toy oclif integration (#214)
This commit adds [oclif] as a command-line framework. It is successfully
integrated with webpack.

[oclif]: https://github.com/oclif/oclif

Usage:
`yarn backend` to build the cli.
`node bin/sourcecred.js` to launch the CLI and see usage
`node bin/sourcecred.js example` for one example command
`node bin/sourcecred.js goodbye` for another example command
2018-05-04 19:28:37 -07:00
Dandelion Mané e3469f157d
Add `src/tools/bin/printCombinedGraph.js` (#207)
`printCombinedGraph` loads and prints a cross-plugin combined
contribution graph for a given GitHub repository.

It is a simple executable wrapper around `src/tools/loadCombinedGraph`.

Example usage:
`node bin/printCombinedGraph.js sourcecred example-git $GITHUB_TOKEN`
2018-05-04 12:10:20 -07:00
Dandelion Mané e66ed45cba
Add CLI for printing a fresh Git graph (#206)
`cloneAndPrintGitGraph` clones a git repository, and generates a Git
object graph for that repository.

This can be run as follows:
```
yarn backend;
node bin/cloneAndPrintGitGraph sourcecred example-git
```

This commit also adds two utility modules:
* `cloneAndLoadRepository` , which clones a Git repository to a tmpdir,
parses the `Repository` data out, and then cleans up.
* `cloneGitGraph`, which calls `cloneAndLoadRepository` and `createGraph`

Test plan: These don't fit well into our CI, because they require
network access to clone repositories from GitHub. I verified that the
functions work via the demo script above.
2018-05-04 11:35:14 -07:00
William Chargin f3a440244e
Fix all lint errors, adding a lint CI step (#175)
Test Plan:
Run `yarn lint` and `yarn travis` and observe success. Add something
that triggers a lint warning, like `const zzz = 3;`; re-run and observe
failures.

wchargin-branch: lint
2018-04-30 14:52:28 -07:00
William Chargin 1c28c75e39
Check in example repo’s in-memory representation (#166)
Summary:
Two reasons for this. First, we want tests to be able to operate on this
data without having to generate repositories via `git(1)`. (Doing that
is slow, and requires a Git installation, and makes it less clear that
the tests are correctly isolated/provides more surface area for
something to go wrong.) Second, in general plugins will need a canonical
source of test data, so setting/continuing this precedent is a good
thing.

Test Plan:
Observe that the old Jest snapshot must be equivalent to the new JSON
one, because the test criterion in `loadRepository.test.js` changed and
the test still passes. Then, run `loadRepositoryTest.sh` and note that
it passes; change the `example-git.json` file and note that the test
fails when re-run; then, run the test with `--updateSnapshot` and watch
it magically revert your changes.

wchargin-branch: check-in-git-repo
2018-04-27 20:51:54 -07:00
William Chargin d6e9b0a72b
Add a command-line script to create example repos (#155)
Summary:
We’ll use this to create the repositories on disk and then push them to
GitHub.

Test Plan:
Generate both kinds of repository, and check out the SHAs:
```shell
$ yarn backend
$ node bin/createExampleRepo.js /tmp/repo
$ node bin/createExampleRepo.js --submodule /tmp/repo-submodule
$ node bin/createExampleRepo.js --no-submodule /tmp/repo-no-submodule
$ # (first and third lines do the same thing)
$ git -C /tmp/repo rev-parse HEAD
677b340674bde17fdaac3b5f5eef929139ef2a52
$ git -C /tmp/repo-submodule rev-parse HEAD
29ef158bc982733e2ba429fcf73e2f7562244188
$ git -C /tmp/repo-no-submodule rev-parse HEAD
677b340674bde17fdaac3b5f5eef929139ef2a52
```
Then, note that these SHAs are expected per the snapshot file in
`exampleRepo.test.js.snap`.

wchargin-branch: create-example-repo-command
2018-04-26 19:53:46 -07:00
Dandelion Mané 39fd3fa354
Make GitHub capitalization consistent within code (#100)
* Make GitHub capitalization consistent within code

We now never capitalize the H in GitHub within variable or function
names. We still capitalize it in comments or user facing strings.

Test plan:
Unit tests, the fetchGithubRepoTest.sh, and
`git grep itHub` only shows comment lines and print statements.

* Fix William's klaxon
2018-03-20 18:32:05 -07:00
William Chargin bbecf00615
Repurpose React app as artifact editor (#89)
Summary:
We’ll now start creating the artifact plugin. A large part of this will
be the user interface, including a GUI. For now, our build system just
builds a single React app, so we’re cannibalizing the main explorer to
serve this purpose.

Paired with @dandelionmane.

Test Plan:
The following still work:
  - `yarn test`
  - `yarn start`
  - `yarn build; (cd build; python -m SimpleHTTPServer)`

wchargin-branch: repurpose-react-app-as-artifact-editor
2018-03-19 15:25:23 -07:00
William Chargin ca85fdf234 Reorganize `src/` directory (#87)
Test Plan:
Note that tests still pass, and all changes to snapshot files are
verbatim moves.

wchargin-branch: reorg
2018-03-19 14:31:50 -07:00
William Chargin 274007c90d
Configure Webpack for backend applications (#84)
Summary:
Running `yarn backend` will now bundle backend applications. They’ll be
placed into the new `bin/` directory. This enables us to use ES6 modules
with the standard syntax, Flow types, and all the other goodies that
we’ve come to expect. A backend build takes about 2.5s on my laptop.

Created by forking the prod configuration to a backend configuration and
trimming it down appropriately.

To test out the new changes, this commit changes `fetchGitHubRepo` and
its driver to use the ES6 module system and Flow types, both of which
are properly resolved.

Test Plan:
Run `yarn backend`. Then, you can directly run an entry point via
```
$ node bin/fetchAndPrintGitHubRepo.js sourcecred example-repo "${TOKEN}"
```
or invoke the standard test driver via
```shell
$ GITHUB_TOKEN="${TOKEN}" src/backend/fetchGitHubRepoTest.sh
```
where `${TOKEN}` is your GitHub authentication token.

wchargin-branch: webpack-backend
2018-03-18 22:43:23 -07:00
Dandelion Mané bc2377448f
Move package json to root (#37)
Reorganize the code so that we have a single package.json file, which is at the root.
All source code now lives under `src`, separated into `src/backend` and `src/explorer`.

Test plan:

- run `yarn start` - it works
- run `yarn test` - it finds the tests (all in src/explorer) and they pass
- run `yarn flow` - it works. (tested with an error, that works too)
- run `yarn prettify` - it finds all the js files and writes to them
2018-02-26 22:32:23 -08:00