This commit updates eslint from v4 to v6. In doing so, I've moved off of
the create-react-app base eslint config. We were on an old version (v2)
and it doesn't make sense to update to v4, as in v4 create-react-app
uses typescript. Also, it didn't make sense to stay on
create-react-app's v2 config, because then it had unmet peer dependency
constraints on old versions of eslint.
Instead, I've moved us to use the default rules for eslint,
eslint-plugin-react, and eslint-plugin-flowtype.
I also made some changes to the codebase to satisfy the new lint rules
that came with this change.
Test plan: `yarn test` passes.
This commit updates our prettier version from `1.13` to `1.18`. Looks
like software does get better over time! I like all of the changes.
Test plan: `yarn test` passes. I've manually inspected the diffs.
When we took a dep on better-sqlite3 in #836, we used a fork, because
better-sqlite3 did not yet support private in-memory databases via the
`:memory:` filepath. As of better-sqlite3 v5, this has been added to
mainline, so we no longer need the fork.
The v4->v5 transition involves some breaking changes. The only ones that
affected us were two field renames, from `lastUpdateROWID` to
`lastUpdateRowid`, and `returnsData` to `reader`.
Test plan:
After updating the field accesses, `yarn test --full` passes. For added
safety, I also blew away cache, loaded a nontrivial repository, and
verified that the full cred workflow still works.
cc @wchargin
Resolves#1054
Added "ROCKET" and "EYES" to the list of reaction types.
Added "ROCKET" as a valid cred signal, kept "EYES" invisible.
My approach was to use `git git grep THUMBS_UP '*.js'`
and `git grep ThumbsUp '*.js'` to find all the relevant files,
as suggested in #1054
**Test Plan**
1) Inspecting Sourcecred/Mission's UI:
[#13] contains: GOT 🚀 FROM 1 user
@BrianLitwin contains: REACTED 🚀 TO 1 issue
@BrianLitwin contains: REACTED 🚀 TO #13
2) Yarn Test passes
3) `github/edges.test` includes a snapshot test to verify
that we can create an edge using ROCKET
4) @wchargin also noted that:
```sh
diff -u <(git grep -c 'THUMBS_UP' '*.js') <(git grep -c 'ROCKET' '*.js')
diff -u <(git grep -c 'ThumbsUp' '*.js') <(git grep -c 'Rocket' '*.js')
```
passes.
`graphql/mirror.test` now includes "ROCKET" and "EYES" in the example
GithubSchema, but their inclusion has no effect
on any tests.
**Screenshots**
1.
<img width="378" alt="screenshot 2019-01-22 09 02 12" src="https://user-images.githubusercontent.com/26695477/51540428-6c87b600-1e24-11e9-8334-1d9d993dce01.png">
2.
<img width="525" alt="screenshot 2019-01-22 09 02 41" src="https://user-images.githubusercontent.com/26695477/51540472-84f7d080-1e24-11e9-8847-245c0c09ddd6.png">
<br>
Shoutout to [this comment], which saved me an untold amount of head-scratching,
and also @Decentralion's help debugging in the Issue thread.
[#13]: https://github.com/sourcecred/mission/issues/13
[this comment]: e0762303d4/src/plugins/github/graphqlTypes.test.js (L13-L15)
Summary:
This was used for ad hoc testing of the Mirror module before it was
integrated into SourceCred. We haven’t kept it up to date with schema
changes, and it is no longer needed: you can just run `sourcecred load`.
This was also the only untested code in the `graphql/` package, so it is
nice to remove it.
Test Plan:
Running `yarn test --full` passes.
wchargin-branch: remove-mirror-demo
Summary:
This enables us to deal with GraphQL remotes that violate their contract
guarantees and provide a node of the wrong type. An instance in which
the GitHub GraphQL API does this is documented here:
<https://gist.github.com/wchargin/a2b8561b81bcc932c84e493d2485ea8a>
A Mirror cache is only valid for a fixed set of blacklisted IDs. This is
necessary to ensure consistency. If the set of blacklisted IDs changes,
simply remove the existing database and download again from scratch.
Test Plan:
Unit tests added, with full coverage.
wchargin-branch: mirror-blacklist
Summary:
We have a `const Reactions` convenience enum in `github/graphql.js`.
That value is useful, but that module is slated to die. This commit
extends our Flow type generation script to include these values.
Test Plan:
Existing unit tests suffice.
wchargin-branch: schema-generate-enums
Summary:
Generating Flow types from a structured schema is both straightforward
and terribly useful. This commit implements it.
Test Plan:
Inspect the snapshot for correctness manually. Then, copy it into a new
file, remove backslash-escapes, and verify that it passes Flow.
A subsequent commit will generate types for the actual GitHub schema.
wchargin-branch: graphql-generate-flow-types
Summary:
Prior to this change, primitive fields were un\[i\]typed. This commit
allows adding type annotations. Such annotations do not change the
semantics at all, but we will be able to use them to generate Flow types
corresponding to a schema.
This commit also strengthens the validation on schemata to catch some
errors that would previously have gone unnoticed at schema construction
time: e.g., a node reference to a type that does not exist.
Test Plan:
Unit tests updated, retaining full coverage. The demo script continues
to work when loading `example-github` or `sourcecred`.
wchargin-branch: schema-annotated-primitives
Summary:
This commit follows up on the previous two pull requests by drawing the
rest of the owl.
Resolves#915.
Test Plan:
Unit tests included.
To verify the snapshot change, open the snapshot file, and copy
everything from `query TestUpdate {` through the matching `}`, not
including the enclosing quotes. Strip all backslashes (Jest adds them).
Post the resulting query to GitHub and verify that it completes
successfully and that the result contains a commit with an `author`.
In other words, `xsel -ob | tr -d '\\' | ghquery | jq .` with [ghquery].
[ghquery]: https://gist.github.com/wchargin/630e03e66fa084b7b2297189615326d1
The demo entry point has also been updated. For an end-to-end test, you
can run the following command to see a commit with a `null` author (with
the current state of the repository) and a commit with a non-`null`
author:
```
$ node bin/mirror.js /tmp/mirror-example.db \
> Repository MDEwOlJlcG9zaXRvcnkxMjMyNTUwMDY= \
> 3600 2>/dev/null |
> jq '(.defaultBranchRef.target, .pullRequests[0].mergeCommit) | {url, author}'
{
"url": "6bd1b4c0b7",
"author": {
"date": "2018-09-12T19:48:21-07:00",
"user": null
}
}
{
"url": "0a223346b4",
"author": {
"date": "2018-02-28T00:43:47-08:00",
"user": {
"id": "MDQ6VXNlcjE0MDAwMjM=",
"__typename": "User",
"url": "https://github.com/decentralion",
"login": "decentralion"
}
}
}
```
You can also check that it is possible to fetch the whole SourceCred
repository (ID: `MDEwOlJlcG9zaXRvcnkxMjAxNDU1NzA=`).
wchargin-branch: mirror-shallow
Summary:
See #915 for context. This adds nested field data to the “useful info”
data structure added in #857.
Test Plan:
Unit tests for `_buildSchemaInfo` updated.
wchargin-branch: mirror-schemainfo-shallow
Summary:
See #915 for context. This commit changes the `schema` module only.
I had a hard time picking names that clearly distinguish the top-level
field on the object and the subfields that it contains. @decentralion
and I independently came up with “nest” and “egg”. It’s a bit colorful,
but it’s certainly easy to remember which one is which, and it doesn’t
conflict with existing notions like “parent”/“child”.
Test Plan:
Unit tests expanded slightly, retaining full coverage.
wchargin-branch: schema-shallow
Summary:
This completes the minimum viable public API for the `Mirror` class. As
described on the docstring, the typical workflow for a client is:
- Invoke the constructor to acquire a `Mirror` instance.
- Invoke `registerObject` to register a root object of interest.
- Invoke `update` to update all transitive dependencies.
- Invoke `extract` to retrieve the data in structured form.
It is the third step that is added in this commit.
In this commit, we also include a command-line demo of the Mirror
module, which exercises exactly the workflow above with a hard-coded
GitHub schema. This can be used to test the module’s behavior with
real-world data. I plan to remove this entry point once we integrate
this module into SourceCred.
This commit makes progress toward #622.
Test Plan:
Unit tests included for the core functionality. The imperative shell
does not have automated tests. You can test it as follows.
First, run `yarn backend` to build `bin/mirror.js`. Then, run:
```shell
$ node bin/mirror.js /tmp/mirror-demo.db \
> Repository MDEwOlJlcG9zaXRvcnkxMjMyNTUwMDY= \
> 60
```
Here, the big base64 string is the ID for the sourcecred/example-github
repository. (This is listed in `graphql/demo.js`, alongside the ID for
the SourceCred repository itself.) The value 60 is a TTL in seconds. The
database filename is arbitrary.
This will print out a ton of output to stderr (all intermediate queries
and responses, for debugging purposes), and then the full contents of
the example repository to stdout.
Run the command again, and it should finish instantly. On my machine,
the main function runs faster than the Node startup time (50ms vs 60ms).
Then, re-run the command, changing the TTL from `60` to `1`. Note that
it sends off some queries and then prints the same repository.
It is safe to kill the program at any time, either while waiting for a
response from GitHub or while processing the results, because the mirror
module takes advantage of SQLite’s transaction-safety. Intermediate
updates will be persisted, so you’ll lose just a few seconds of
progress.
You can also of course dive into the generated database yourself to
explore the data. It’s good fun.
wchargin-branch: mirror-e2e-update
Summary:
GitHub has an undocumented node limit on the number of IDs that can be
provided to the top-level `nodes` connection. This is silly, because we
can just spread the IDs over multiple identical connections. This commit
implements the logic to do so.
Test Plan:
Create some queries that use `nodes(ids: ...)` to fetch varying numbers
of objects:
```shell
id="MDEwOlJlcG9zaXRvcnkxMjAxNDU1NzA="
nodes() {
n="$1"
ids="$(yes "$id" | head -n "$n" | jq -R . | jq -sc .)"
printf 'nodes(ids: %s) { __typename }' "$ids"
}
query() {
printf '{ '
for num; do
printf 'nodes_%s: %s ' "$num" "$(nodes "$num")"
done
printf '}'
}
```
Note that the query given by `query 101` results in an error…
```json
{
"data": null,
"errors": [
{
"message": "You may not provide more than 100 node ids; you provided 101.",
"type": "ARGUMENT_LIMIT",
"path": [
"nodes_101"
],
"locations": [
{
"line": 1,
"column": 3
}
]
}
]
}
```
…but the query given by `query 98 99` happily returns 197 node entries.
wchargin-branch: mirror-paginate-own-data
Summary:
An update step is now roughly as simple as:
_updateData(postQuery(_queryFromPlan(_findOutdated())))
(with some option/config parameters thrown in).
This makes progress toward #622.
Test Plan:
Unit tests included. They are light, because these functions are light.
They still retain full coverage.
wchargin-branch: mirror-full-pipeline
Summary:
These typenames are often superfluous, but sometimes they are useful.
For instance, we might want to fetch the same data for `User`s, `Bot`s,
and `Organization`s, but still differentiate which kind of node we
fetched from an `Actor` union reference. Similarly, many timeline events
may have similar signatures (like, “issue closed” vs. “issue reopened”).
Test Plan:
Existing unit tests have been updated; run `yarn unit`.
wchargin-branch: mirror-extract-typenames
Summary:
The `extract` method lets you get data out of a mirror in a structured
format.
The mirror module now contains all the plumbing needed to provide
meaningful value. Remaining to be implemented are some internal
porcelain and a public method to perform an update step.
This makes progress toward #622.
Test Plan:
Comprehensive unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-extract
Summary:
This reestablishes harmony in light of #882.
Test Plan:
Existing unit tests suffice; run `yarn unit`.
wchargin-branch: mirror-query-plan-connection-object-typename
Summary:
This commit adds internal functions to (a) emit a GraphQL query to fetch
data for own-data of an object, and (b) ingest the results of said query
back into the database.
The API and implementation differ from the connection-updating analogues
introduced in #878 in that the query for own data is independent of an
object’s ID: it depends only on the object’s type. This affords us more
flexibility in composing queries.
As described in a internal documentation comment, values are stored in
the database in JSON-stringified form: we cannot use the obvious raw SQL
values, because there is no native encoding of booleans (`0`/`1` is
conventional), and we need to distinguish them from other data types.
There are other ways to solve this problem. Notably:
1. We could take inspiration from OCaml: encode stronger static types
and a simpler runtime representation. That is, we could change the
schema field types from simply “primitive” to the various specific
primitive types. Then, when reading data out from the database, we
could reinterpret the values appropriately.
2. We could take advantage of the fact that we are not using all of
SQLite’s data types. In particular, we do not store anything as a
binary blob, so we could encode `false` as a length-0 zeroblob and
`true` as a length-1 zeroblob, for instance. Again, when reading
data out from the database, we would reinterpret the values—but in
this approach we would not need an explicit schema from the user.
For now, we take the easiest and simplest approach just to get ourselves
off the ground. We can easily move to the second option described above
later.
This commit makes progress toward #622.
Test Plan:
Unit tests included, with full coverage. While these tests check that
the GraphQL queries are as expected, they cannot check that they are
actually valid in production. To check this, follow the instructions in
the added snapshot test.
wchargin-branch: mirror-own-data-updates
Summary:
As <https://github.com/sourcecred/sourcecred/pull/883/files#r219648511>.
It is somewhat unfortunate that this mixes a command with a query, but
the concession is acceptable in this instance, I think.
Test Plan:
Existing unit tests suffice, retaining full coverage.
wchargin-branch: mirror-register-node-field-results
Summary:
This helpful utility is already used in some test code, and will shortly
be used in main code. @decentralion suggested factoring it out in
<https://github.com/sourcecred/sourcecred/pull/883#discussion_r219647781>.
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-make-update-helper
Summary:
Almost every GitHub connection has nodes of an object type, like `User`
or `IssueComment`. But a few have nodes of union type, including
`IssueTimelineItemConnection` (which we will likely want to query), and
those require special handling. This commit adds susupport for such
connections.
Analysis to determine which connections have non-object elements:
<https://gist.github.com/wchargin/647fa7ed8d6d17ae2e204bd098104407>
Test Plan:
Unit tests modified appropriately, retaining full coverage.
The easiest way to verify the snapshot is probably to copy the raw
contents (everything inside the quotes) into `/tmp/snapshot`, then run:
```shell
$ sed -e 's/\\//g' </tmp/snapshot >/tmp/query # Jest adds backslashes
$ jq -csR '{query: ., variables: {}}' </tmp/query >/tmp/payload
$ ENDPOINT='https://api.github.com/graphql'
$ AUTH="Authorization: bearer ${SOURCECRED_GITHUB_TOKEN}"
$ curl "$ENDPOINT" -X POST -H "$AUTH" -d @- </tmp/payload >/tmp/result
```
and then execute the JQ program mentioned in the comment in the test
case, and verify that it prints `true`.
wchargin-branch: connection-of-union
Summary:
This commit changes the `Issue` type of the schema used in the `mirror`
tests to have fields a faithful subset of those in the actual GitHub
schema. The tests are self-contained, so this is not strictly required.
However, it is convenient, because it means that we can snapshot a query
that can actually be posted to GitHub.
Test Plan:
Running `yarn unit mirror` suffices for the code change. The GitHub
schema docs at <https://developer.github.com/v4/object/issue/> indicate
that each of `id`, `url`, `author`, `repository`, `title`, and
`comments` is a valid field.
wchargin-branch: mirror-test-schema
Summary:
This commit adds internal functions to (a) emit a GraphQL query to fetch
data for a particular connection, and (b) ingest the results of said
query back into the database.
This commit makes progress toward #622.
Test Plan:
Unit tests included, with full coverage. While these tests check that
the GraphQL queries are as expected, they cannot check that they are
actually valid in production. To check this, follow the instructions in
the added snapshot test.
wchargin-branch: mirror-connection-updates
Summary:
This function finds all objects whose own data has not been updated
since a given time, and all connections whose entries have not been
updated since that time.
Note that this is scoped to the entirety of the database. In #622,
I discussed using a recursive common table expression to identify only
those transitive dependencies of the root. I think that this is overkill
for the `_findOutdated` method: you’ll usually want to update everything
in the database. Don’t worry—the cool recursive query will still be used
in the `extract` function. :-)
This commit makes progress toward #622.
Test Plan:
Unit tests added, with full coverage; run `yarn unit`.
wchargin-branch: mirror-findoutdated
Summary:
This function informs the GraphQL mirror of the existence of an object,
specified by its global ID and its concrete typename (“concrete” meaning
“object type”—like `User`, not `Actor`).
The function will be called extensively internally as more objects are
discovered while traversing the graph, but also needs to be exposed as a
public entry point: a client needs to call this function at least once
to register the root node of interest. A typical client workflow, once
all of #622 is implemented, might be:
1. Issue a standalone GraphQL query to find the ID of a root node, like
a GitHub repository: `repository(owner: "foo", name: "bar") { id }`.
2. Call `registerObject` with the ID found in the previous step.
3. Instruct the mirror to recursively update all dependencies.
4. Extract data from the mirror.
As of this commit, steps (1) and (2) are possible.
This commit makes progress toward #622.
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-registerobject
Summary:
It’s useful to add this simple function now because the rest of the
commits required to implement #622 will want to use it extensively in
test code. Actual clients of the API will not need to use it, because
the concept of “updates” is an implementation detail: clients will
always provide simple timestamps.
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-createupdate
Summary:
Some clients want to write
const primitivesTableName = _primitivesTableName(typename);
which they cannot if the function is also called `primitivesTableName`,
due to ECMAScript shadowing semantics.
Test Plan:
Running `yarn flow` suffices; running `yarn unit` really suffices.
wchargin-branch: mirror-rename-primitivestablename
Summary:
Each change provides real value, by either testing a plausible happy
path that simply was not tested previously, or by adding an
`empty`-assertion to a switch against a discriminated union type.
Test Plan:
For the snapshot change relating to the query formatter, note that
Prettier formats the changed portion of the snapshot in the same way, by
visiting <https://prettier.io/playground> and setting the parser to
"graphql". (Prettier in general agrees with the stringification defined
by this module, except for commas and spacing, for which we don’t bother
to generate impeccably pretty output.)
Run `yarn coverage` and note that the coverage of the whole `graphql`
package is 100% on all axes.
wchargin-branch: graphql-coverage
Summary:
This simplifies and clarifies the code with no observable change.
Test Plan:
Existing unit tests suffice; run `yarn unit`.
wchargin-branch: mirror-use-schemainfo
Summary:
This is mostly useful not for computational efficiency, but for ease of
implementation: there end up being multiple places where we want to find
(say) the primitive fields on an object, and having to go through the
whole iterate-and-switch-and-push process repeatedly is annoying.
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-schema-info
Summary:
This commit augments the `Mirror` constructor to turn the provided
GraphQL schema into a SQL schema, with which it initializes the backing
database. The schema is roughly as originally described in #622, with
some changes (primarily: we omit `WITHOUT ROWID`; we add indexes; we
store `total_count` on connections; and we use milliseconds instead of
seconds for epoch time).
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-sql-schema
Summary:
This commit introduces the `Mirror` class that will be the centerpiece
of the persistent-loading API as described in #622. An instance of this
class represents a mirror of a remote GraphQL database, defined by a
particular schema. In this commit, we add the construction logic, which
includes a safety measure to ensure that the database is used within one
version of the code and schema.
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-class
Summary:
In implementing #622, we’ll want to run lots of things inside of
transactions. This commit introduces a JavaScript API to do so more
easily, properly handling success and failure cases.
Test Plan:
Unit tests included, with full coverage; run `yarn unit`.
wchargin-branch: mirror-transaction-helper
Summary:
This affords more flexibility to clients, because an exact value can be
used in place of an inexact value, but not vice versa.
Test Plan:
Running `yarn flow` suffices.
wchargin-branch: schema-exact-type-fields
Summary:
GraphQL unions are required to be unions specifically of object types.
They cannot contain primitives or other union types as clauses. This is
good: it means that we don’t have to worry about unions that recursively
reference each other or themselves.
Unions are also required to have at least one clause, but we don’t
validate this because it’s not helpful for us. An empty union is
perfectly well-defined, if useless, and shouldn’t cause any problems.
Relevant portion of the spec:
<https://facebook.github.io/graphql/October2016/#sec-Union-type-validation>
Test Plan:
Unit tests added, retaining full coverage; `yarn unit` suffices.
wchargin-branch: graphql-schema-union-validation
Summary:
This commit introduces a module for declaratively specifying the schema
of a GraphQL database. See `buildGithubSchema` in `schema.test.js` for
an example of the API.
This makes progress toward #622, though the design has evolved some
since its original specification there.
Test Plan:
Unit tests added, with full coverage; `yarn unit` suffices.
wchargin-branch: graphql-schema
Summary:
Per #800, each test file should start with a `describe` block listing
its file path under `src`. Currently, nine of our tests do not do so.
Of these, eight had a top-level describe block with the wrong name
(either not a filepath or an outdated filepath), while only one short
test was missing a top-level describe block altogether. This patch fixes
each file to use the correct format.
Test Plan:
Apply the Sharness test in #802, and note that it fails before this
patch but passes after it.
wchargin-branch: describe-fix
Our data model orients on getting repos from GitHub, which are
alternatively represented as strings like "sourcecred/sourcecred", or
pairs of variables representing the owner and name, or objects with
owner and name properties. We also have a few different implementations
of repo validation, which are not applied consistently.
This commit changes all that. We now have a consistent Repo type which
is an object containing a string owner and string name. Thanks to a
clever suggestion by @wchargin, it is implemented as an opaque subtype
of an object containing those properties, so that the only valid way to
construct a Repo typed object is to use one of the functions that
consistently validates the repo.
As a fly-by fix, I noticed that there were some functions in the GitHub
query generation that didn't properly mark arguments as readOnly. I've
fixed these.
Test plan: No externally-observable behavior changes (except insofar as
there is a slight change in variable names in the GitHub graphql query,
which has also resulted in a snapshot diff). `yarn travis --full`
passes. `git grep repoOwner` presents no hits.
Test plan:
`git grep -i v3` only shows incidental hits in longer strings
`yarn travis --full` passes
`yarn backend` works
`yarn build` works
`yarn start` works
`node bin/sourcecred.js start` works
`node bin/sourcecred.js load sourcecred example-github` works
Paired with @wchargin
We want to reset some of our basic assumptions, and make `Graph` into a
pure graph implementation, rather than a hybrid graph and key-value
store.
This is a substantial rewrite, so we want to start from scratch in a v3/
directory and pull code into v3 as necessary. So that we can do this in
a relatively clean fashion, we're first moving the v1 and v2 code into
their own directories.
Paired with @wchargin
Test plan: Travis, and `yarn backend`, `node bin/sourcecred.js start`.
Note that `yarn backend` and `node bin/sourcecred.js start` both use the
v1 versions. We'll migrate those (by changing paths.js) to v3 when
appropriate.
Test Plan:
Run `yarn lint` and `yarn travis` and observe success. Add something
that triggers a lint warning, like `const zzz = 3;`; re-run and observe
failures.
wchargin-branch: lint
Summary:
For pagination, we’ll want to query against multiple entities of the
same type. GraphQL uses aliases to facilitate this. This commit adds
support for aliases to our GraphQL query DSL.
Test Plan:
Inspect snapshot changes, and note that `yarn flow` and `yarn test`
pass.
wchargin-branch: graphql-aliases
