Commit Graph

1162 Commits

Author SHA1 Message Date
William Chargin a5e0707cb9
ui: remove `:visited` color change for nav links (#892)
Test Plan:
Note that the nav links are now a lighter color, except when `:active`
(e.g., when you’re holding down the mouse button).

wchargin-branch: remove-visited
2018-09-25 16:38:46 -07:00
William Chargin a7f29cb057
mirror: add connection object types to query plan (#891)
Summary:
This reestablishes harmony in light of #882.

Test Plan:
Existing unit tests suffice; run `yarn unit`.

wchargin-branch: mirror-query-plan-connection-object-typename
2018-09-25 14:25:11 -07:00
William Chargin 892c498f9c
mirror: query and process own-data updates (#883)
Summary:
This commit adds internal functions to (a) emit a GraphQL query to fetch
data for own-data of an object, and (b) ingest the results of said query
back into the database.

The API and implementation differ from the connection-updating analogues
introduced in #878 in that the query for own data is independent of an
object’s ID: it depends only on the object’s type. This affords us more
flexibility in composing queries.

As described in a internal documentation comment, values are stored in
the database in JSON-stringified form: we cannot use the obvious raw SQL
values, because there is no native encoding of booleans (`0`/`1` is
conventional), and we need to distinguish them from other data types.
There are other ways to solve this problem. Notably:

 1. We could take inspiration from OCaml: encode stronger static types
    and a simpler runtime representation. That is, we could change the
    schema field types from simply “primitive” to the various specific
    primitive types. Then, when reading data out from the database, we
    could reinterpret the values appropriately.

 2. We could take advantage of the fact that we are not using all of
    SQLite’s data types. In particular, we do not store anything as a
    binary blob, so we could encode `false` as a length-0 zeroblob and
    `true` as a length-1 zeroblob, for instance. Again, when reading
    data out from the database, we would reinterpret the values—but in
    this approach we would not need an explicit schema from the user.

For now, we take the easiest and simplest approach just to get ourselves
off the ground. We can easily move to the second option described above
later.

This commit makes progress toward #622.

Test Plan:
Unit tests included, with full coverage. While these tests check that
the GraphQL queries are as expected, they cannot check that they are
actually valid in production. To check this, follow the instructions in
the added snapshot test.

wchargin-branch: mirror-own-data-updates
2018-09-25 11:58:32 -07:00
William Chargin 2af8566e6a
app: add and use a shared Link component (#890)
Summary:
This will enable us to style links consistently across our application.

Our previous link colors for base and `:visited` were so similar that I
didn’t actually realize that they were different. In this change, I’ve
kept the same base color, and selected a more contrasting `:visited`
color. I also added an `:active` color, which is good for usability
(color chosen via <http://paletton.com/>’s “triad” suggestion).

Example screenshot, with active, visited, and base links:
![Screenshot as in this commit][img-underline]

I also considered implementing the link underlines with `border-bottom`
instead of `text-decoration` (an oft-touted suggestion that has always
smelled fishy to me, but [the W3 does it][w3], so I guess it’s okay).
That would look like this:

![Screenshot with `border-bottom` underlines][img-border]

…but I did not do so ([rationale in a comment on #890][rationale]).

[w3]: https://www.w3.org/TR/WCAG20-TECHS/F73.html
[img-underline]: https://user-images.githubusercontent.com/4317806/45987381-f154f580-c025-11e8-8b0f-63c1e1ddce02.png
[img-border]: https://user-images.githubusercontent.com/4317806/45926176-a081b780-bed4-11e8-96f2-d0d24d11c8f7.png
[rationale]: https://github.com/sourcecred/sourcecred/pull/890#issuecomment-424146773

We can certainly change these decisions later—that’s one of the purposes
of having this abstraction—so I’m not inclined to bikeshed on them too
much in this commit.

Implementation adapted from:
<80263b190e/src/components/Link.js>

Test Plan:
Check that `<a>` elements and React Router link elements are used only
in `Link` and snapshots:

```
$ git grep --name-only -Fw '<a'
src/app/Link.js
src/assets/logo/discourse_512.png
src/plugins/github/__snapshots__/render.test.js.snap
$ git grep '"react-router"' | grep 'Link'
src/app/Link.js:import {Link as RouterLink} from "react-router";
src/app/Link.test.js:import {Link as RouterLink} from "react-router";
src/app/createRelativeHistory.test.js:import {Router, Route, Link} from "react-router";
src/app/withAssets.test.js:import {IndexRoute, Link, Router, Route} from "react-router";
```

Check that the primary color now appears in just one spot:

```
$ git grep -i 0872a2
src/app/Link.js:    ...colorAttributes("#0872A2"),
```

Then, run `yarn start` and click all the links. Note in particular that
the SVG icons in the header have the correct colors in the active state
as well as the base state.

wchargin-branch: app-link
2018-09-24 18:26:52 -07:00
William Chargin 3257df63fe
mirror: add helper to register nullable node results (#889)
Summary:
As <https://github.com/sourcecred/sourcecred/pull/883/files#r219648511>.
It is somewhat unfortunate that this mixes a command with a query, but
the concession is acceptable in this instance, I think.

Test Plan:
Existing unit tests suffice, retaining full coverage.

wchargin-branch: mirror-register-node-field-results
2018-09-24 18:20:28 -07:00
William Chargin 28c4e497fb
mirror: factor out `_makeSingleUpdateFunction` (#888)
Summary:
This helpful utility is already used in some test code, and will shortly
be used in main code. @decentralion suggested factoring it out in
<https://github.com/sourcecred/sourcecred/pull/883#discussion_r219647781>.

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-make-update-helper
2018-09-24 18:06:14 -07:00
William Chargin 252400d5e7
flow: fix `Statement` properties (#886)
Summary:
These were completely wrong; my bad.

[The docs][1] list `source` and `returnsData` but not `database`.
I include the latter only for consistency with `Transaction`, and will
consider removing them both as they are technically undocumented (though
not underscore-named, so it’s not clear-cut).

[1]: https://wchargin.github.io/better-sqlite3/api.html#properties-1

Test Plan:
At a Node console:

```js
> require("better-sqlite3")(":memory:").prepare("BEGIN")
Statement {
  returnsData: false,
  source: 'BEGIN',
  database:
   Database {
     inTransaction: false,
     open: true,
     memory: false,
     readonly: false,
     name: ':memory:' } }
```

wchargin-branch: flow-better-sqlite3-statement-properties
2018-09-21 17:48:28 -07:00
Dandelion Mané 3b962eacea
Git: track the repositories containing each commit (#884)
This modifies the Git `Repository` data structure so that for every
commit, we track the `RepoId`s of repos containing that commit. This way
we will be able to do things like hyperlink to the right url for that
commit.

`loadRepository` has been modified to set the initial `repoId`.
`mergeRepository` has been updated to ensure that it concatenates the
`repoId`s properly.
Tests were added for both cases.

The example-git snapshot has been updated accordingly.

Test plan: `yarn test --full`
2018-09-21 17:06:14 -07:00
Dandelion Mané b506d40efd
Add the `RepoIdString` type (#885)
This modifies `core/repoId` so that `repoIdToString` returns not a
`string`, but an opaque subtype of `string` called `RepoIdString`.

This allows us to store stringified `RepoId`s (which is useful whenever
we want value-not-reference semantics, like for use as a map key)
while still maintaining a type assertion that the strings, in fact,
represent valid `RepoId`s.

Test plan: A unit test verifies that it's a flow error to cast a string
to a `RepoIdString`.
2018-09-21 16:54:20 -07:00
William Chargin 838092194b
mirror: add support for connections of union types (#882)
Summary:
Almost every GitHub connection has nodes of an object type, like `User`
or `IssueComment`. But a few have nodes of union type, including
`IssueTimelineItemConnection` (which we will likely want to query), and
those require special handling. This commit adds susupport for such
connections.

Analysis to determine which connections have non-object elements:
<https://gist.github.com/wchargin/647fa7ed8d6d17ae2e204bd098104407>

Test Plan:
Unit tests modified appropriately, retaining full coverage.

The easiest way to verify the snapshot is probably to copy the raw
contents (everything inside the quotes) into `/tmp/snapshot`, then run:

```shell
$ sed -e 's/\\//g' </tmp/snapshot >/tmp/query  # Jest adds backslashes
$ jq -csR '{query: ., variables: {}}' </tmp/query >/tmp/payload
$ ENDPOINT='https://api.github.com/graphql'
$ AUTH="Authorization: bearer ${SOURCECRED_GITHUB_TOKEN}"
$ curl "$ENDPOINT" -X POST -H "$AUTH" -d @- </tmp/payload >/tmp/result
```

and then execute the JQ program mentioned in the comment in the test
case, and verify that it prints `true`.

wchargin-branch: connection-of-union
2018-09-21 16:20:58 -07:00
Dandelion Mané 23ae7e2f08
Rename the core `Repo` type to `RepoId` (#881)
We have a core type called `Repo`, but it really is an identifier to a
repo, rather than being a repo itself. This is confusing since we have a
data type called `Repository` which actually represents the data in a
repository. I've renamed `Repo` to `RepoId` for clarity.

Test plan: `yarn test --full` passes. Running the frontend passes after
wholly regenerating the sourcecred data directory.
2018-09-21 14:44:29 -07:00
Dandelion Mané 1e5f728e29
Cred explorer: display commit short hash + summary (#879)
This modifies how commits are displayed in the cred explorer. Rather
than printing the full hash, we now print a short hash followed by the
summary.

Test plan:
Snapshot is updated, also I tested it by running SourceCred on a real
repository.
2018-09-21 13:24:28 -07:00
William Chargin 09ed51ed6e
mirror: align test schema more with GitHub schema (#880)
Summary:
This commit changes the `Issue` type of the schema used in the `mirror`
tests to have fields a faithful subset of those in the actual GitHub
schema. The tests are self-contained, so this is not strictly required.
However, it is convenient, because it means that we can snapshot a query
that can actually be posted to GitHub.

Test Plan:
Running `yarn unit mirror` suffices for the code change. The GitHub
schema docs at <https://developer.github.com/v4/object/issue/> indicate
that each of `id`, `url`, `author`, `repository`, `title`, and
`comments` is a valid field.

wchargin-branch: mirror-test-schema
2018-09-20 17:17:27 -07:00
Dandelion Mané 0b3e2e7f7f
Load commits' short hash and summary (#876)
This modifies the Git Commit type to includea short hash and a oneline
summary, and modifies `loadRepository` so that we actually get that
data.

The example-git repository has been updated to include a commit with
leading whitespace and a pipe in the summary, to ensure that these are
respected.

Test plan: Observe that the snapshot is updated, and the updates are
correct. `yarn test --full` passes.
2018-09-20 15:51:58 -07:00
William Chargin 6ae5c56624
mirror: query and process connection updates (#878)
Summary:
This commit adds internal functions to (a) emit a GraphQL query to fetch
data for a particular connection, and (b) ingest the results of said
query back into the database.

This commit makes progress toward #622.

Test Plan:
Unit tests included, with full coverage. While these tests check that
the GraphQL queries are as expected, they cannot check that they are
actually valid in production. To check this, follow the instructions in
the added snapshot test.

wchargin-branch: mirror-connection-updates
2018-09-20 15:46:03 -07:00
Dandelion Mané 5348fe68bf
Remove git adresses for trees, blobs, and entries (#877)
In #873 I removed the data types for trees, blobs, and entries, but
neglected to remove the address related code. This commit corrects that
mistake. Some test cases in other modules have been removed because the
failure is now structurally impossible, e.g. it is not possible that we
would provide a non-commit address to the GitHub plugin, because
non-commit addresses do not exist.

Test plan: `yarn test --full` passes.
2018-09-20 15:18:01 -07:00
William Chargin 1dd8b7bcb7
mirror: add internal method `_findOutdated` (#875)
Summary:
This function finds all objects whose own data has not been updated
since a given time, and all connections whose entries have not been
updated since that time.

Note that this is scoped to the entirety of the database. In #622,
I discussed using a recursive common table expression to identify only
those transitive dependencies of the root. I think that this is overkill
for the `_findOutdated` method: you’ll usually want to update everything
in the database. Don’t worry—the cool recursive query will still be used
in the `extract` function. :-)

This commit makes progress toward #622.

Test Plan:
Unit tests added, with full coverage; run `yarn unit`.

wchargin-branch: mirror-findoutdated
2018-09-20 14:19:47 -07:00
Dandelion Mané e3f04c5079
Git plugin serializes the repository and graph (#874)
This modifies the behavior when loading the Git plugin so that it
serializes the Repository as well as the graph. This will allow us to
get extra information, like the commit headline, to the Git plugin in
the frontend.

As an added bonus, we can now refactor `loadRepositoryTest` to depend on
`sourcecred.js load` rather than `loadAndPrintRepository`. As this was
the only use for `loadAndPrintRepository`, we can safely delete it. This
improves our test quality because it means we are also testing the
actual CLI behavior.

Note that the switch from using `stringify` to `json.tool` for
pretty-printing has resulted in a trivial diff in the snapshot.

Test plan: `yarn test --full` passes.
2018-09-20 14:07:58 -07:00
Dandelion Mané 5b8ec53b13
Remove all git tree related code (#873)
In #627, I made a case for removing all trees and blobs from the cred
graph. The issue was that the data was bloated and noisy, and did not
provide much value in its current form. This commit follows on that by
actually removing the code from the codebase (rather than keeping it
unused).

I want to delete this code because I believe:
1. It is unlikely to see further use in its current form (because
collecting the entire Git tree structure is just too noisy for our
purposes)
2. In the event that we do need it, reviving it will not be too
difficult (because it is all quite locally scoped to the Git plugin).
3. Keeping unused code increases ongoing maintenance + development
costs, and I'd like to bias towards keeping the codebase simple and
lean.

In the event that a future contributor is reviving this code and finds
it a pain, I pre-emptively apologize to you.

Test plan:
`yarn test --full` passees.
2018-09-20 13:50:27 -07:00
William Chargin ab5b6ecb68
mirror: add public method `registerObject` (#870)
Summary:
This function informs the GraphQL mirror of the existence of an object,
specified by its global ID and its concrete typename (“concrete” meaning
“object type”—like `User`, not `Actor`).

The function will be called extensively internally as more objects are
discovered while traversing the graph, but also needs to be exposed as a
public entry point: a client needs to call this function at least once
to register the root node of interest. A typical client workflow, once
all of #622 is implemented, might be:

 1. Issue a standalone GraphQL query to find the ID of a root node, like
    a GitHub repository: `repository(owner: "foo", name: "bar") { id }`.
 2. Call `registerObject` with the ID found in the previous step.
 3. Instruct the mirror to recursively update all dependencies.
 4. Extract data from the mirror.

As of this commit, steps (1) and (2) are possible.

This commit makes progress toward #622.

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-registerobject
2018-09-20 13:03:06 -07:00
Dandelion Mané 12aa5b7439
Add `mergeRepository` for merging Git repositories (#871)
This commit adds a utility method, `mergeRepository`, which can merge
multiple Git repository data structures. `loadGitData` has been updated
to create a merged repository and then subsequently generate a graph
from it.

Test plan:
New unit tests were added. `yarn test --full` passes. Loading a project
and viewing its git data in the cred explorer works.
2018-09-20 12:57:57 -07:00
Dandelion Mané 8442255db3
Remove obsolete eslint TODOs (#872)
I wanted to add two new flowtype lint rules. However, as seen in #853
and #854, these rules don't work very well. As such, I'm giving up on
those two and removing the TODOs.

Test plan: n/a
2018-09-20 12:52:07 -07:00
William Chargin 7702bb2e1d
flow: simplify better-sqlite3 bound parameters (#869)
Summary:
The actual constraints for bound parameters are too complicated to
express within Flow. This commit changes the type definitions from one
approximation to another, simpler one. Neither approximation is likely
to cause many problems in practice, either in terms of spurious errors
or spurious lacks of error. (The failure mode for the new formulation is
having multiple dictionaries of binding values, which would pass Flow
but quickly raise a `TypeError` at runtime.)

The reason for the change is that it makes the method definitions
considerably simpler, in a way that is likely to avoid other problems
with Flow. In particular, this removes method overloads and the need for
parameter disambiguation.

I fix a typo while in the area.

Test Plan:
Note that the following file typechecks:

```js
// @flow
import Database from "better-sqlite3";
const db = new Database(":memory:");
const stmt = db.prepare("BEGIN"); // SQL text doesn't matter

stmt.run();
stmt.run(null, 2, "three", new Buffer(Array(4)));
stmt.run(+false);
stmt.run(1, {dos: 2}, 3); // the binding dictionary can go anywhere
stmt.run({a: 1}, {b: 2}); // this will fail at runtime (TypeError)

// $ExpectFlowError
stmt.run(false); // booleans cannot be bound
// $ExpectFlowError
stmt.run({x: {y: "z"}}); // named parameters are not recursive
```

All but the last two success cases (lines 9 and 10) would also have
passed before this change.

wchargin-branch: flow-better-sqlite3-bound-parameters-simplify
2018-09-20 11:31:26 -07:00
William Chargin e572551cd8
mirror: add internal method `_createUpdate` (#868)
Summary:
It’s useful to add this simple function now because the rest of the
commits required to implement #622 will want to use it extensively in
test code. Actual clients of the API will not need to use it, because
the concept of “updates” is an implementation detail: clients will
always provide simple timestamps.

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-createupdate
2018-09-20 11:12:21 -07:00
Dandelion Mané cdceedef8d
Display urls in the cred explorer (#860)
This commit modifies the plugin adapter's `nodeDescription` method so
that it may return a React node.

This enables the GitHub plugin's `nodeDescription` method to include
hyperlinks directly to the referenced content on GitHub. This makes
examining e.g. comment cred much easier.

I've also made two other changes to the descriptions:
- Pull requests diffs now color-encode the additions and deletions
- Descriptions for comments and reviews no longer include the authors

The Git plugin's behavior is unchanged.

Test plan:
I loaded a large repository in the cred explorer and verified that
exploring comments and pulls and issues is much easier. The descriptions
are as expected for every category of node. Snapshot tests updated.

Fixes #590.
2018-09-20 10:48:05 -07:00
William Chargin ac46a98a0b
travis: pin Node v8.x.x (#867)
Summary:
SourceCred officially supports Node v8.x.x, so Travis should be testing
against that version of Node.

This fixes a cron-only, Travis-only failure. The test for the build
script asserts that stderr only contains some known expected messages.
On Node v10, which Travis was using, there is an additional deprecation
message due to <https://github.com/yarnpkg/yarn/issues/5477>.

Test Plan:
To see that this fixes the current cron-only build failure, add the
`--full` argument to the `test` script in `package.json`, and push the
resulting tree to Travis. Note that it runs `sharness-full`, which
passes.

wchargin-branch: travis-node-8
2018-09-19 18:18:34 -07:00
William Chargin 778ce9eb93
prettier: ignore sharness/ (#866)
Summary:
Sharness tests use temporary directories under the `sharness/`
directory, which sometimes contain build output, which includes
JavaScript files. This creates a flaky failure in `yarn test`: if
`sharness` creates a file then `check-pretty` can complain about it.

Test Plan:
None.

wchargin-branch: prettier-ignore-sharness
2018-09-19 18:12:38 -07:00
William Chargin 678962ba7a
travis: fix notification email config (#864)
Summary:
In #339, we attempted to configure Travis to email us on build failure.
But we misread [the docs]: the emails need to be preceded by a hyphen.

[the docs]: https://docs.travis-ci.com/user/notifications#configuring-email-notifications

This is understandable because the Travis documentation renders the
critical hyphen with `#999` against a background of `#f5f2f0` at 12px,
which has a contrast ratio of only 2.55, failing WCAG 2.0 categorically.

Test Plan:
Fingers crossed.

wchargin-branch: travis-email-config
2018-09-19 17:07:15 -07:00
William Chargin 8b22c1edd5
flow: change better-sqlite3 `each` to `iterator` (#863)
Summary:
This was changed in the v4.0.0 release of `better-sqlite3`:
<https://github.com/JoshuaWise/better-sqlite3/pull/61>

Test Plan:
Create the following test file, and note that it fails to typecheck
before this change (on two counts) but passes after:

```js
// @flow
import Database from "better-sqlite3";
const db = new Database(":memory:");
db.prepare("CREATE TABLE foo (id)").run();
const insert = db.prepare("INSERT INTO foo (id) VALUES (?)");
insert.run(1);
insert.run(2);
const retrieve = db.prepare("SELECT id FROM foo").pluck();

console.log("Old way:");
try {
  // $ExpectFlowError
  retrieve.each((x) => void console.log(x));
} catch (e) {
  console.log("Failed (good):", String(e));
}
console.log();

console.log("New way:");
for (const value of retrieve.iterate()) {
  console.log(value);
}
```

Also, run the test with `NODE_ENV=development babel-node src/test.js`,
and note that it outputs:

```
$ NODE_ENV=development babel-node src/test.js
Old way:
Failed (good): TypeError: retrieve.each is not a function

New way:
1
2
```

wchargin-branch: flow-better-sqlite3-iterator
2018-09-19 15:39:01 -07:00
William Chargin 5debae414e
mirror: rename helper `_primitivesTableName` (#861)
Summary:
Some clients want to write

    const primitivesTableName = _primitivesTableName(typename);

which they cannot if the function is also called `primitivesTableName`,
due to ECMAScript shadowing semantics.

Test Plan:
Running `yarn flow` suffices; running `yarn unit` really suffices.

wchargin-branch: mirror-rename-primitivestablename
2018-09-19 14:28:26 -07:00
William Chargin 1d18652459
graphql: improve coverage of `queries` module (#859)
Summary:
Each change provides real value, by either testing a plausible happy
path that simply was not tested previously, or by adding an
`empty`-assertion to a switch against a discriminated union type.

Test Plan:
For the snapshot change relating to the query formatter, note that
Prettier formats the changed portion of the snapshot in the same way, by
visiting <https://prettier.io/playground> and setting the parser to
"graphql". (Prettier in general agrees with the stringification defined
by this module, except for commas and spacing, for which we don’t bother
to generate impeccably pretty output.)

Run `yarn coverage` and note that the coverage of the whole `graphql`
package is 100% on all axes.

wchargin-branch: graphql-coverage
2018-09-18 17:57:16 -07:00
William Chargin 85efa811e0
mirror: use `SchemaInfo` in `_initialize` (#858)
Summary:
This simplifies and clarifies the code with no observable change.

Test Plan:
Existing unit tests suffice; run `yarn unit`.

wchargin-branch: mirror-use-schemainfo
2018-09-18 16:30:43 -07:00
William Chargin e69ff57c58
mirror: precompute some useful schema info (#857)
Summary:
This is mostly useful not for computational efficiency, but for ease of
implementation: there end up being multiple places where we want to find
(say) the primitive fields on an object, and having to go through the
whole iterate-and-switch-and-push process repeatedly is annoying.

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-schema-info
2018-09-18 16:24:38 -07:00
William Chargin 1b1a1e4d46
mirror: embed GraphQL schema into SQL (#849)
Summary:
This commit augments the `Mirror` constructor to turn the provided
GraphQL schema into a SQL schema, with which it initializes the backing
database. The schema is roughly as originally described in #622, with
some changes (primarily: we omit `WITHOUT ROWID`; we add indexes; we
store `total_count` on connections; and we use milliseconds instead of
seconds for epoch time).

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-sql-schema
2018-09-18 13:31:34 -07:00
Dandelion Mané 786a38e773
Improve CI performance by limiting max workers (#856)
According to the [jest docs], setting `maxWorkers=4` can substantially
improve performance on CI.

This commit sets `maxWorkers=4` when running `yarn unit` as a part of
`yarn test`. Based on my local testing (see data below), this improves
performance locally in addition to the expected performance improvement
on travis.

\## Testing `yarn unit` by itself
```
yarn unit --maxWorkers=1
15.28s user 3.71s system 112% cpu 16.848 total
15.13s user 3.68s system 112% cpu 16.708 total
15.32s user 3.76s system 112% cpu 16.917 total
15.91s user 3.74s system 112% cpu 17.425 total
15.61s user 3.76s system 113% cpu 17.125 total

yarn unit --maxWorkers=2
19.43s user 4.03s system 212% cpu 11.061 total
19.86s user 4.19s system 210% cpu 11.407 total
21.19s user 4.26s system 213% cpu 11.902 total
20.68s user 4.51s system 212% cpu 11.873 total
20.78s user 4.26s system 212% cpu 11.780 total

yarn unit --maxWorkers=4
29.43s user 5.14s system 389% cpu 8.865 total
29.99s user 5.18s system 392% cpu 8.961 total
32.22s user 5.52s system 390% cpu 9.659 total
33.99s user 5.97s system 395% cpu 10.097 total
33.38s user 5.93s system 395% cpu 9.933 total

yarn unit --maxWorkers=8
48.21s user 6.57s system 621% cpu 8.815 total
51.61s user 7.16s system 610% cpu 9.622 total
59.48s user 7.82s system 621% cpu 10.833 total
58.18s user 8.10s system 624% cpu 10.607 total
58.92s user 8.22s system 620% cpu 10.817 total

unset
46.27s user 6.44s system 599% cpu 8.799 total
49.08s user 7.04s system 600% cpu 9.342 total
54.85s user 7.52s system 600% cpu 10.383 total
55.66s user 7.52s system 605% cpu 10.438 total
53.77s user 7.50s system 604% cpu 10.142 total
```

\## Testing `yarn test`
```
maxWorkers=1
46.65s user 5.92s system 249% cpu 21.038 total
47.94s user 5.81s system 251% cpu 21.354 total
51.50s user 6.44s system 260% cpu 22.234 total
52.60s user 6.65s system 268% cpu 22.077 total
53.04s user 6.27s system 266% cpu 22.278 total

maxWorkers=2
56.13s user 6.13s system 409% cpu 15.204 total
63.32s user 7.22s system 412% cpu 17.091 total
64.82s user 7.19s system 422% cpu 17.027 total
64.59s user 7.41s system 417% cpu 17.227 total
65.40s user 7.30s system 419% cpu 17.318 total

maxWorkers=4
74.64s user 7.60s system 584% cpu 14.066 total
82.69s user 8.43s system 582% cpu 15.643 total
85.00s user 8.68s system 591% cpu 15.835 total
84.81s user 8.58s system 595% cpu 15.690 total
85.22s user 8.59s system 596% cpu 15.719 total

maxWorkers=4 and everything depends on unit
59.29s user 6.01s system 378% cpu 17.261 total
62.99s user 6.64s system 375% cpu 18.564 total
65.54s user 7.31s system 375% cpu 19.419 total
63.24s user 7.13s system 379% cpu 18.548 total
63.68s user 7.13s system 383% cpu 18.457 total

maxWorkers=8
92.85s user 8.13s system 643% cpu 15.702 total
101.63s user 9.21s system 632% cpu 17.510 total
101.63s user 9.23s system 636% cpu 17.428 total
101.81s user 9.32s system 633% cpu 17.546 total
101.62s user 9.39s system 632% cpu 17.542 total

unset
88.75s user 8.15s system 646% cpu 14.988 total
96.43s user 9.23s system 631% cpu 16.739 total
98.27s user 9.17s system 638% cpu 16.819 total
98.46s user 9.01s system 642% cpu 16.729 total
98.53s user 9.15s system 637% cpu 16.889 total

unset + everything depends on unit
76.02s user 7.61s system 486% cpu 17.208 total
79.14s user 8.26s system 484% cpu 18.030 total
84.32s user 9.19s system 488% cpu 19.136 total
84.92s user 9.14s system 497% cpu 18.919 total
84.46s user 8.94s system 492% cpu 18.965 total
```

Test plan: `yarn test` passes here and on travis

[jest docs]: https://jestjs.io/docs/en/troubleshooting
2018-09-18 13:17:51 -07:00
Dandelion Mané f03e3c83f2
Fix a crash when a review comment has >5 reactions (#855)
Our continuation-fetching code failed to properly get continuations for
pull request review comments, because it was only asking for more
reactions on `"IssueComment"` fragments. This caused the
`ensureNoMorePages` function to properly throw an error rather than
proceding with incomplete data.

This commit fixes the root cause by splitting
`continuationsFromComment`into `continuationsFromReviewComment` and
`continuationsFromIssueComment`. (Pull and issue comments are both
considered 'IssueComment's.) The example-github repository has been
updated to include 10 reactions to a single review comment; the
example-data was updated in this commit, and all reactions have been
loaded.

I've also added a `console.error` statement in `ensureNoMorePages`. This
only triggers when the program is about to fail, and it's useful for
debugging.

Test plan: `yarn test --full` passes.

Paired with @wchargin
2018-09-18 12:52:05 -07:00
William Chargin fa81c4eaa9
git: properly load empty repositories (#851)
Summary:
Fixes #850.

Test Plan:
Regression test added; it fails before the change and passes after it.
Also, running `node ./bin/sourcecred.js load wchargin/mt` (which is a
GitHub repository with no commits) now successfully loads the
repository. (The cred explorer fails to process it, because it tries to
normalize across GitHub users, of which there are none, but this is a
known limitation and is unrelated.)

wchargin-branch: fix-empty-git-repository
2018-09-17 16:36:22 -07:00
Dandelion Mané 7259233f82
Prepare to enable flow-type eslint rules (#848)
This commit upgrades the flow-type eslint plugin to latest, and writes
new rules into the eslintrc. To keep the diff clean, the rules are
disabled: I will turn them on individually (fixing errors) in followon
commits.

Test plan: `yarn test`.
Uncommenting the lines produces many lint errors (but the linter still operates as expected).
2018-09-17 14:11:39 -07:00
William Chargin a93ad80ebc
mirror: initialize a GraphQL database mirror (#847)
Summary:
This commit introduces the `Mirror` class that will be the centerpiece
of the persistent-loading API as described in #622. An instance of this
class represents a mirror of a remote GraphQL database, defined by a
particular schema. In this commit, we add the construction logic, which
includes a safety measure to ensure that the database is used within one
version of the code and schema.

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-class
2018-09-17 13:53:08 -07:00
Dandelion Mané 62d3c180ee
Add GitHub reactions to the graph (#846)
* Define Reaction edges

This adds support to `github/edges` for creating edges representing
GitHub reactions. These edges are not actually added to the graph.

Test plan: Unit tests

* Add GitHub reactions to the graph

This commit adds functional support for reactions in SourceCred.
Only thumbs-up, heart, and hooray reactions are supported for now, as
they are all unambiguously positive; adding support for negative
reactions like thumbs-down will require some more thought.

The reactions are added to the graph, and new edge types have been added
to the UI.

Test plan:
The `graphView` class has been updated to do invariant checking for the
reaction edges, including that the unsupported reaction types like
"THUMBS_DOWN" aren't added to the graph.

I've tested this feature by downloading data for a large repository
(ipfs/go-ipfs). The reaction edges appear and transfer cred reasonably.
The edge types are displayed in the weight config appropriately.

Builds on #839, #840, and #845.
2018-09-17 13:44:11 -07:00
Dandelion Mané 488c98c3e1
Add tensorflow-gardener to the list of bots (#841)
Test plan: I verified that the spelling is correct
2018-09-17 13:44:02 -07:00
Dandelion Mané 33d14b9d1a
Define Reaction edges (#845)
This adds support to `github/edges` for creating edges representing
GitHub reactions. These edges are not actually added to the graph.

Test plan: Unit tests
2018-09-17 13:35:47 -07:00
William Chargin e9279bee90
mirror: add a helper function for transactions (#844)
Summary:
In implementing #622, we’ll want to run lots of things inside of
transactions. This commit introduces a JavaScript API to do so more
easily, properly handling success and failure cases.

Test Plan:
Unit tests included, with full coverage; run `yarn unit`.

wchargin-branch: mirror-transaction-helper
2018-09-17 13:33:10 -07:00
William Chargin f966ce300f
schema: make `fields` and `clauses` exact (#843)
Summary:
This affords more flexibility to clients, because an exact value can be
used in place of an inexact value, but not vice versa.

Test Plan:
Running `yarn flow` suffices.

wchargin-branch: schema-exact-type-fields
2018-09-17 12:07:52 -07:00
Dandelion Mané 1ad2cc0958
Request reactions data from GitHub (#839) (#840)
This commit updates the GitHub graphql query to also fetch reactions.
We update the JSON typedefs to include this new information, add
continuations from comments, and update existing continuation and query
code. Also, I added a safety check when updating comments for issues
that was previously unnecessary but is now needed.

Test plan:
- `yarn test --full` passes.
- Setting the page limits to 1 and running on the example-github does
not error with unexhausted pages, and loads all the expected reactions.
- Running on a larger repository (go-ipfs) works as expected.
- I have written dependent code that consumes these reactions in the
RelationalView, and works as intended, which suggests that the type
signatures are correct.
2018-09-17 11:47:37 -07:00
William Chargin 51461f4842
flow: mark bound parameters as covariant (#842)
Summary:
Before this patch, an object whose type has read-only attributes cannot
be passed to `stmt.run`/etc., because the libdef does not promise not to
mutate its argument. This patch fixes that oversight.

Test Plan:
Create the following test file, and note that it fails to typecheck
before this change but passes after:

```js
// @flow
import Database from "better-sqlite3";
const db = new Database("/tmp/foo");
const args: {|+id: number|} = {id: 1};
db.prepare("INSERT INTO foo (id) VALUES (?)").run(args);
```

wchargin-branch: flow-better-sqlite3-bound-parameters-covariant
2018-09-17 11:15:04 -07:00
Dandelion Mané a2ffdf5ca8
Request reactions data from GitHub (#839)
This commit updates the GitHub graphql query to also fetch reactions.
We update the JSON typedefs to include this new information, add
continuations from comments, and update existing continuation and query
code. Also, I added a safety check when updating comments for issues
that was previously unnecessary but is now needed.

Test plan:
- `yarn test --full` passes.
- Setting the page limits to 1 and running on the example-github does
not error with unexhausted pages, and loads all the expected reactions.
- Running on a larger repository (go-ipfs) works as expected.
- I have written dependent code that consumes these reactions in the
RelationalView, and works as intended, which suggests that the type
signatures are correct.
2018-09-14 16:10:31 -07:00
Dandelion Mané aecf64b026
Detect references to commits (#833)
Now that #832 gave us logic to parse references to commits, we have the
RelationalView find and add these references. The actual change is
a simple extension of existing reference detection logic.

Test plan: Observe that the snapshots are updated with references to
commits from the example-github repository.

Progress on #815.
2018-09-14 11:56:16 -07:00
William Chargin c616ec82fb
flow: add type definitions for `better-sqlite3` (#837)
Summary:
The `flow-typed` repository doesn’t have type definitions for
`better-sqlite3`, so I wrote some. I figure that we can use them for a
while, fix any problems that arise, and then PR them upstream.

I started from [the TypeScript definitions][1], but made some
improvements (like stronger typing for bound parameters), and also made
some necessary changes due to differences between Flow and TypeScript.

[1]: https://github.com/DefinitelyTyped/DefinitelyTyped/commits/master/types/better-sqlite3/index.d.ts

Prettier does not format this file (it is in `flow-typed`), so I
manually ran it through Prettier with the settings used by `flow-typed`
itself.

Test Plan:
None.

wchargin-branch: flow-better-sqlite3
2018-09-13 18:32:00 -07:00
Dandelion Mané c9a0d4b1b8
Modify parseReferences to detect refs to commits (#832)
We add a new function, `findCommitReferences`, which can find both
explicit url references to commits, and commit hashes.

Since the commit url includes the commit hash, some extra logic is added
to deduplicate them in this instance. Tests verify that this is done
properly.

Test plan: Unit tests cover the cases of having commit hashes, having
commit urls, and having both at once.
2018-09-13 18:21:06 -07:00