Commit Graph

67 Commits

Author SHA1 Message Date
William Chargin ca85fdf234 Reorganize `src/` directory (#87)
Test Plan:
Note that tests still pass, and all changes to snapshot files are
verbatim moves.

wchargin-branch: reorg
2018-03-19 14:31:50 -07:00
Dandelion Mané 30600004e4
Implement the GitHub graph parser (#81)
The GitHub parser transforms GraphQL api data from GitHub into our Graph
data structure. This commit focuses on properly parsing Issues, Pull
Requests, Comments, and Users.

Test Plan:
Run the unit tests. Inspect the snapshot results (particuarly those for
individual pull requests or issues, which are easier to parse) and
verify that the output is appropriate.
2018-03-19 12:26:32 -07:00
William Chargin 274007c90d
Configure Webpack for backend applications (#84)
Summary:
Running `yarn backend` will now bundle backend applications. They’ll be
placed into the new `bin/` directory. This enables us to use ES6 modules
with the standard syntax, Flow types, and all the other goodies that
we’ve come to expect. A backend build takes about 2.5s on my laptop.

Created by forking the prod configuration to a backend configuration and
trimming it down appropriately.

To test out the new changes, this commit changes `fetchGitHubRepo` and
its driver to use the ES6 module system and Flow types, both of which
are properly resolved.

Test Plan:
Run `yarn backend`. Then, you can directly run an entry point via
```
$ node bin/fetchAndPrintGitHubRepo.js sourcecred example-repo "${TOKEN}"
```
or invoke the standard test driver via
```shell
$ GITHUB_TOKEN="${TOKEN}" src/backend/fetchGitHubRepoTest.sh
```
where `${TOKEN}` is your GitHub authentication token.

wchargin-branch: webpack-backend
2018-03-18 22:43:23 -07:00
William Chargin 291dcb17c3
Add a GraphQL structured query format (#77)
Summary:
See motivation in #76. Feel free to look at the new snapshot file to
inspect the structured representation and also the stringified output.

This implementation is sufficient to encode our query against the
GitHub v4 API; see the test plan below.

Test Plan:
Unit tests added; run `yarn flow && yarn test`.

This code has full coverage except for lines 260, 315, and 380 of
`queries.js`; these lines check invariants that should never be
violated.

You can also use the following steps to verify that the sample query is
valid GraphQL that produces the same results as our hand-written query:

 1. Apply the following hacky patch:

    ```diff
    diff --git a/src/backend/graphql/queries.test.js b/src/backend/graphql/queries.test.js
    index 52bdec7..c04a636 100644
    --- a/src/backend/graphql/queries.test.js
    +++ b/src/backend/graphql/queries.test.js
    @@ -3,6 +3,18 @@
     import type {Body} from "./queries";
     import {build, stringify, multilineLayout, inlineLayout} from "./queries";

    +function emitGitHubQuery(layout, filename) {
    +  const fs = require("fs");
    +  const path = require("path");
    +  const result = stringify.body(usefulQuery(), layout);
    +  const outputFilepath = path.join(__dirname, "..", filename);
    +  const outputText = `module.exports = ${JSON.stringify(result)};\n`;
    +  fs.writeFileSync(outputFilepath, outputText);
    +  console.log(`Wrote output to ${outputFilepath}.`);
    +}
    +emitGitHubQuery(multilineLayout("  "), "githubQueryMultiline.js");
    +emitGitHubQuery(inlineLayout(), "githubQueryInline.js");
    +
     describe("queries", () => {
       describe("end-to-end-test cases", () => {
         const testCases = {
    ```

 2. Run `CI=true yarn test`, and verify that the following two files
    written to `src/backend/` contain appropriate contents. You can just
    eyeball them, or check that they match my results:
    https://gist.github.com/wchargin/f37b99fd4ec345c9d2541c2dc53ceda9

 3. In `fetchGitHubRepo.js`, change the definition of `const query` to

    ```js
    const query = require("./githubQueryMultiline.js");
    ```

    Run

    ```shell
    GITHUB_TOKEN="<your_token_here>" src/backend/fetchGitHubRepoTest.sh
    ```

    and verify that it exits successfully.

 4. Repeat for `require("./githubQueryInline.js")`.

wchargin-branch: graphql-structured-queries
2018-03-18 22:35:20 -07:00
William Chargin 1083540d21
Parameterize `Graph` over node and edge payloads (#83)
Summary:
Closes #82. This affords clients type-safety without needing to
verbosely annotate every node or edge passed into the graph functions.
It also enables graph algorithms to be more expressive in their types:
for instance, the merge function now clearly indicates from its type
that the first graph’s nodes are passed as the first argument to the
node reducer, and the second graph’s nodes to the second. Clients can
upgrade immediately by using `Graph<*, *>`.

Thankfully, Flow supports variance well enough for this all to be
possible without too much trouble.

Test Plan:
Existing unit tests pass statically and at runtime. I added a test case
to demonstrate that merging works covariantly.

To see some failures, change `string` to an incompatible type, like
`number`, in the definitions of `makeGraph` in test functions for
conservatively rejecting graphs with conflicting nodes/edges
(ll. 446, 462).

wchargin-branch: parameterize-graph
2018-03-17 11:36:53 -07:00
Dandelion Mané 894d6a2291
Allow adding explicitly typed nodes/edges (#80)
Summary:
Flow doesn’t allow us to specify variance annotations in generic
function parameters, and doesn’t allow coercing `Node<T>` to
`Node<mixed>`. This forces us to put `any`s in our code, which…works.

Paired with @dandelionmane.

Test Plan:
New unit tests trivially pass dynamically, and now pass statically
(failing before the changes to `graph.js`).

wchargin-branch: explicitly-typed-nodes-edges
2018-03-17 00:28:33 -07:00
Dandelion Mané 1e791782d5
Allow redundant adds to the Graph (#79)
Graph.addNode and Graph.addEdge now allow adding the same node or edge
multiple times, provided that the duplicate adds are trying to insert
identical content.

This came up while prototyping the GitHub plugin; rather than create
myriad subgraphs and merge them, I found it convenient to construct a
single graph and iteratively add nodes. Since the same node may be
discovered multiple times (most notably user identities), there was a
need for a "conservative add" abstraction that adds a node if it doesn't
exist yet, but errors only if multiple adds conflict.

Since this behavior is generic and highly conservative, it seemed
appropriate to include in the graph class itself.

Test Plan:
The unit tests have been updated to include the new behavior.
2018-03-17 00:28:17 -07:00
Dandelion Mané d2501947a6
Regenerate example-repo testdata (#78)
I moved sourcecred/tiny-example-repository to sourcecred/example-repo
as it's simpler to remember. I also unarchived it and added comments
to an issue, so that we can create a simple test for issue parsing.

This commit merely updates SourceCred to point to example-repo with
the regenerated canoncial output.
2018-03-16 11:22:17 -07:00
Dandelion Mané 0e57b42095 Fetch GitHub repos using the GitHub v4 API (#75)
Summary:
It’s a whole new world of GraphQL! Our parser is now just a GraphQL
query that asks for exactly what we want and dumps it to a file. The
data exposed by the v4 API is also in a much nicer format than that of
the v3 API, so this is pretty much a universal improvement.

Currently, we do not handle pagination. We require that the repository
in question have fewer than a fixed number of issues, and comments per
issue, and reviews per PR, and review comments per PR, and so on. If
this limit is exceeded, the script will fail-fast with a nice error
message. To fix this, we’ll need to write a general-purpose pagination
API that allows traversing cursors at any level of the query.

Paired with @wchargin.

Test Plan:
Run

    $ GITHUB_TOKEN="your_token_here" src/backend/fetchGitHubRepoTest.sh

and verify that it exits with 0. Note that if you change this script’s
repository from `tiny-example-repository` to `sourcecred`, the script
correctly fails and outputs a useful diff.

wchargin-branch: github-v4-graphql
2018-03-15 14:56:25 -07:00
Dandelion Mané ef7160d7d4
Update README.md
Guide contributors to participate via issues.
2018-03-06 19:09:46 -08:00
Dandelion Mané fb00c35823
Factor eventide graph demo data to a new module (#71)
* Factor evertide graph demo data to a new module

It would be helpful to make our standard tiny graph available to other
test and demo instances, outside of just graph.test.js. This way we can
use it as a test case for the Graph Explorer.
2018-03-05 20:58:47 -08:00
Dandelion Mané 7ea8bdd964
Make App.js into skeleton for GraphExplorer (#70)
* Make App.js into skeleton for GraphExplorer

We make a very basic skeleton for the Graph Explorer as a basis
for future development.

This commit also removes the UserExplorer and FileExplorer from
App.js. Since we have changed the underlying data model, we are
unlikely to use the UserExplorer or FileExplorer in anything like
their current state, so they are effectively deprecated. I am deferring
removing them because it is nice to have some examples of working React
code to copy from, before the Graph Explorer is ready.

Test plan: run `yarn start`, and observe that the App displays the
words "Graph Explorer" underneath the "SourceCred Explorer" title bar.
2018-03-05 20:09:24 -08:00
William Chargin 5960eab6c1
Make `Graph` serializable (#69)
Summary:
This commit adds `toJSON()` and `static fromJSON()` on `Graph`. The main
benefit at this time is that this gets us free interoperability with
Jest’s snapshot testing.

The implementation of `fromJSON` is not performance-tuned, and could
probably be significantly optimized.

See #65 for discussion.

Test Plan:
New unit tests added: `yarn flow && yarn test`.

wchargin-branch: make-graph-serializable
2018-03-05 16:22:58 -08:00
William Chargin cee90fd10f
Use `AddressMap` in `Graph` (#68)
Summary:
This commit simplifies the implementation of `Graph` without changing
its interface. We now use the `AddressMap` for all four instance fields
of `Graph`.

Test Plan:
All existing tests pass, and coverage is maintained.

wchargin-branch: use-address-map-in-graph
2018-03-05 16:20:52 -08:00
William Chargin a8da44c94b
Create an `AddressMap` abstraction (#67)
Summary:
This commit reifies the concept of an `Addressable`, which is any object
that has a covariant `address: Address` attribute, and implements a
simple data structure for storing addressable items keyed against their
addresses. Instances of `AddressMap` can replace the four fields of
`Graph`:
```js
_nodes: AddressMap<Node<mixed>>;
_edges: AddressMap<Edge<mixed>>;
_outEdges: AddressMap<{|+address: Address, +edges: Address[]|}>;
_inEdges: AddressMap<{|+address: Address, +edges: Address[]|}>;
```

Test Plan:
New unit tests included, with 100% coverage: `yarn flow && yarn test`.

wchargin-branch: address-map
2018-03-05 16:18:20 -08:00
William Chargin a798f9bac2
Rewrite GitHub plugin payload type system (#64)
Summary:
We’re stripping down the payload types for the GitHub plugin, to only
include what we expect to use immediately. In doing so, we take the
opportunity to make the typing a little stronger, so that we can ensure
that the `type` field of a specific type of payload is set to a
particular constant.

Paired with @dandelionmane.

Test Plan:
Adding these lines to `githubPlugin.js` and running `yarn flow`
indicates that the typechecking is working as expected:
```js
("ISSUE" : NodeType);       // works
("WEIRD" : NodeType);       // fails
("AUTHORSHIP" : EdgeType);  // works
("UNEXPECTED" : EdgeType);  // fails
```

wchargin-branch: github-plugin-payload-types
2018-03-03 15:17:11 -08:00
William Chargin 50c575b2f9
Explicitly test address↔string function error case (#63)
Summary:
`graph.js` coverage is now 100% :-)

Test Plan:
`yarn jest --env=jsdom --coverage` shows no uncovered lines for
`graph.js`, and no failing tests.

wchargin-branch: coverage-gremlin
2018-03-03 15:10:57 -08:00
William Chargin 9b203e8489
Add graph merge functions (#62)
Summary:
Merging graphs will be a common operation. At a per-plugin level, it
will often be useful to build up graphs by creating many very small
graphs and then merging them together. At a cross-project level, we will
need to merge graphs across repositories to gain an understanding of how
value flows among these repositories. It’s important that the core graph
type provide useful functions for merging; this commit adds them.

Test Plan:
New unit tests added; run `yarn flow && yarn test`.

wchargin-branch: graph-merge
2018-03-02 21:35:51 -08:00
William Chargin 82dbf64a2c
Add an equality function for `Graph` (#61)
Summary:
We need this for testing graph equality: deep-equality is not sufficient
because two graphs can be logically equal even if, say, two nodes are
added in different orders.

This commit adds a dependency on `lodash.isequal` for deep equality.

Test Plan:
New unit tests added. Run `yarn flow && yarn test`.

wchargin-branch: graph-equals
2018-03-02 21:13:30 -08:00
William Chargin 5a2380d486
Move `addNode`/`addEdge` tests away from getters (#60)
Summary:
Nothing big; these were just organized wrong.

wchargin-branch: test-reorg
2018-03-02 20:46:35 -08:00
William Chargin 2e5bea2f5e Add Travis badge to README (#59) 2018-03-02 15:06:46 -08:00
Dandelion Mané 6ecf282956
Setup travis CI testing (#58)
Observe that it passed on this commit, but failed builds [#2: Break Tests][2], [#3: Break Flow][3], and [#4: Break Prettier][4]. 

Close #23 

[2]: https://travis-ci.org/sourcecred/sourcecred/builds/348453195
[3]: https://travis-ci.org/sourcecred/sourcecred/builds/348454983
[4]: https://travis-ci.org/sourcecred/sourcecred/builds/348455387
2018-03-02 14:39:54 -08:00
William Chargin 58410c62fa
Replace lingering `mealGraph`s in test case (#57)
Summary:
In merging #54, there was a semantic merge conflict that was not also a
textual merge conflict; this created a failure that only appeared once
that commit was merged.

We propose that to fix this in the future, we only merge commits that
are directly ahead of master.

Test Plan:
This fixes `yarn flow` and `yarn test`.

wchargin-branch: fix-merge-conflict
2018-03-02 14:16:51 -08:00
William Chargin 97446138ab
Make `Address`, `Node`, `Edge` read-only and exact (#56)
Summary:
Again: we assume these invariants, so we may as well encode them.
We should just keep in mind that non-Flow users may wantonly violate
these, so we should still code defensively.

wchargin-branch: readonly-exact
2018-03-02 13:49:34 -08:00
William Chargin f305a48391
Check for `null`/`undefined` in graph functions (#55)
Summary:
These will make nicer error functions in cases where static analysis
doesn’t detect the pollution: e.g., a user isn’t using Flow, or an
expression like `arr[0]` introduces an `undefined`.

Paired with @dandelionmane.

Test Plan:
New unit tests added. Run `yarn test`.

wchargin-branch: null-undefined-check
2018-03-02 13:47:13 -08:00
Dandelion Mané ca3502009b
Create an 'advancedMealGraph' test case (#54)
Create an 'advancedMealGraph' test case

The advancedMealGraph will be a grab-all that holds all advanced and
edge behaviors, e.g. the crab-self-referential loop, and the case
where there are multiple directed edges between the same two nodes.

Aggregating them into one test case will make it easier to test more
complex behaviors, like graph merging and serialization, on the
edge case graphs. However, it's still nice to have the simple graph
so that we can test simple things too. The specific tests for edge
case behavior are left mostly unchanged, in that they start from the
simple graph and add just the advanced feature that they want to test.
2018-03-02 13:45:52 -08:00
William Chargin cae3a92dc9
Add `getAllNodes` and `getAllEdges` functions (#53)
Summary:
Without these functions, it is not possible to meaningfully operate on
an arbitrary graph.

Paired with @dandelionmane.

Test Plan:
New unit tests included. Run `yarn flow && yarn test`.

wchargin-branch: get-all
2018-03-02 11:33:45 -08:00
William Chargin 01510ca63f
Make node and edge types exact (#51)
Summary:
We’ve realized that `u: Edge<T>` implies `u: Node<T>`. That certainly
wasn’t what we were expecting! We might want something like that
eventually, to capture the fact that valuations are themselves valuable,
but for now the type system should encode the assumptions that we’re
actually making. See also #50.

Paired with @dandelionmane.

wchargin-branch: exact-types
2018-03-02 11:31:38 -08:00
Dandelion Mané d26b264e8d
Add "license" field to package.json (#52) 2018-03-02 11:31:12 -08:00
William Chargin 09156bf3f4
Promote `Graph` to a class with useful methods (#49)
Summary:
We had planned to expose our core types as simple Plain Old JavaScript
Objects, with accompanying standalone functions to act directly on these
data structures. We chose this instead of creating `class`es for the
types because it simplifies serialization interop: it obviates the need
for serialization and deserialization functions, because the code is
separated from the data entirely. Reconsidering, we now think that the
convenience benefits of using classes probably outweigh these
serialization cons. Furthermore, this design enables us to separate
ancillary data structures and caches from the raw data, presenting a
cleaner API for consumers of the data.

This commit introduces a `Graph` class and some related logic. With lots
of tests! And 100% code coverage! :-)

Paired with @dandelionmane.

Test Plan:
Run `yarn flow && yarn test` to see the new tests.

wchargin-branch: graph-class
2018-03-01 01:04:11 -08:00
William Chargin f5d486087d
Pull in-edges and out-edges up to top-level graph (#48)
Summary:
The main problem with having these fields on the node is that this
presents the illusion that the API surface area is larger than it
actually is. Clients with reference to a node object could
somewhat-reasonably expect that mutating these fields would be
sufficient to update the structure of the graph, but this isn’t the case
(as the edge objects would need to be updated, too). It’s a nice
semantic bonus, too, as edges aren’t conceptually “part of” nodes.

wchargin-branch: top-level-edges
2018-03-01 00:48:23 -08:00
William Chargin 66243a16c1
Remove weights from the weighted graph (#47)
Summary:
This is an experiment. There are a couple diffferent meanings of
“weight” in play: most prominently, weights assigned by plugins versus
those suitable for comparison among other arbitrary weights. We’re not
sure what the right thing is to put in the actual graph object, so we’re
going to think about this a bit more before adding the field back in.

wchargin-branch: remove-weights
2018-03-01 00:45:52 -08:00
William Chargin 43450f18b1
Rename `sourceId`/`destId` to `src`/`dst` (#46)
Summary:
The “ID” parts were left-over from the Great Address Migration, and we
think that abbreviations are fine here, anyway.

Test Plan:
`yarn flow && yarn test`

wchargin-branch: src-dst-rename
2018-03-01 00:32:30 -08:00
Dandelion Mané cde98cd67b
Remove outdated design documents (#45)
design.md and overview.md both describe a vision of sourcecred
in which it is a measure of credit, and an explicit cryptographic
token. right now, SourceCred is more focused on just measuring credit,
with the expectation that cryptoincentives can be added on later.

Removing these outdated documents will reduce confusion; they may
be re-written and re-added later.
2018-02-28 20:41:04 -08:00
William Chargin 01df727c39
Add tiny-example-repository example data (#44)
Summary:
The sourcecred/tiny-example-repository repository stores some example
data that we can use to generate test cases. As of now, the repository
has been archived so that its state is stable. This commit checks in the
result of our scraper on the repository.

wchargin-branch: example-data
2018-02-28 20:36:18 -08:00
Dandelion Mané 9dc9d5e4f3 Change order of repositoryName and pluginName 2018-02-28 17:47:15 -08:00
Dandelion Mané 58ad1eb635 Add inEdges and outEdges for Nodes. 2018-02-28 17:47:15 -08:00
Dandelion Mané 2992a31157 Graph concept renames
- ID -> Address
- ID.name -> Address.id
- GraphEdge -> Edge
- GraphNode -> Node
2018-02-28 17:47:15 -08:00
William Chargin 791cad9059
Add conversion functions for id (#38)
Test Plan:
Run `yarn test` and note that tests pass.

wchargin-branch: id-conversion
2018-02-26 22:57:00 -08:00
Dandelion Mané bc2377448f
Move package json to root (#37)
Reorganize the code so that we have a single package.json file, which is at the root.
All source code now lives under `src`, separated into `src/backend` and `src/explorer`.

Test plan:

- run `yarn start` - it works
- run `yarn test` - it finds the tests (all in src/explorer) and they pass
- run `yarn flow` - it works. (tested with an error, that works too)
- run `yarn prettify` - it finds all the js files and writes to them
2018-02-26 22:32:23 -08:00
William Chargin c5be6eceda
Create script to scrape data from GitHub repos (#36)
Summary:
This tool grabs all the information that we think will be relevant for a
first-pass implementation of the SourceCred project graph. It includes a
tool to save the results to disk so that we avoid needlessly hitting the
GitHub API over and over.

Paired with @dandelionmane.

Test Plan:
The API doesn’t have tests, because we didn’t think that they would
provide much marginal value. But here’s how you invoke it:

    node bin/fetchAndPrintGitHubRepo.js sourcecred sourcecred "${TOKEN}" >/tmp/out

to crawl the repository `sourcecred/sourcecred` with the given API
token.

wchargin-branch: grab-github-data
2018-02-26 17:11:57 -08:00
Dandelion Mané d41872b7b7
Update README.md (#31)
This reflects our current vision that SourceCred is aimed at valuing open-source contributions via a credit graph, not on directly creating cryptotokens
2018-02-26 17:06:06 -08:00
Dandelion Mané 9878313ef3
Add types for GitHub plugin. (#35) 2018-02-26 17:05:50 -08:00
Dandelion Mané 2c083425fb
Add initial graph data types (#34) 2018-02-26 16:52:24 -08:00
William Chargin d48e8e7f26
Add prettier to backend (#33)
Summary:
Follows the lead of #18.

Test Plan:
Verify that non-pretty files abort the commit:
```shell
$ echo 'const x = 1  +  1;' >foo.js
$ git add foo.js
$ git commit -m yikes
husky > npm run -s precommit (node v8.9.4)

 ❯ Running tasks for **/*.js
   ✖ prettier --list-different
     → foo.js
✖ prettier --list-different found some errors. Please fix them and try committing again.
foo.js

husky > pre-commit hook failed (add --no-verify to bypass)
```

wchargin-branch: backend-prettier
2018-02-26 16:42:53 -08:00
Dandelion Mané 5f55804677
Setup flow for backend. (#32) 2018-02-26 16:40:13 -08:00
Dandelion Mané 4dbb5cd21a
Initial commit of the backend project. (#30)
Sets up package.json, .gitignore, and adds octokit/rest.js as a dependency.
2018-02-26 11:24:00 -08:00
William Chargin 32220148c0 Upgrade: babel-plugin-flow-react-proptypes@^18.0.0 (#27)
Summary:
Generated: `yarn add --dev babel-plugin-flow-react-proptypes@^18.0.0`.

This pulls in brigand/babel-plugin-flow-react-proptypes#182 and
brigand/babel-plugin-flow-react-proptypes#184, both of which affect this
project.

wchargin-branch: upgrade-flow-proptypes-18.0.0
2018-02-18 14:13:42 -08:00
Dandelion Mané b367da8568 Run `yarn prettify` for the first time. (#22) 2018-02-18 08:13:29 -08:00
Dandelion Mané de359dbaa4
Frontend visual cleanup (#21)
* Use font roboto

* Change title.

* Add grid gap

* Restyle app; blue header, and css->inline.

* Further visual cleanup.

- Apply consistent formatting to the two plugin panes.
- Add scroll to the User Explorer.
- Both have white backgrounds with centered titles.
2018-02-18 02:00:11 -08:00