Summary:
We currently load trees and then throw them away later, because we don’t
get useful signal from them. We should consider not doing that. This
will be faster.
Test Plan:
```
$ time node bin/sourcecred.js load tensorflow/tensorflow --plugin git
real 0m33.512s
user 0m35.196s
sys 0m12.489s
```
Also, `yarn test --full` passes.
wchargin-branch: git-deforestation
Adds a link titled "what is this?" that points to my gentle introduction
to cred. Also, move the feedback link to be next to it and get rid of
the prototype disclaimer.
Test plan: Visual inspection, also a test was updated.
Summary:
This fixes a bug where, if the `SOURCECRED_DIRECTORY` environment
variable is set to `foo` but the `-d bar` flag is passed, then the
repository registry will be written under `foo` but the plugin data will
be loaded under `bar`.
Test Plan:
```
$ rm -rf /tmp/good /tmp/bad
$ SOURCECRED_DIRECTORY=/tmp/bad >/dev/null \
> node bin/sourcecred.js load sourcecred/example-github -d /tmp/good
$ [ -d /tmp/bad ]; echo $?
$ find /tmp/good
/tmp/good
/tmp/good/cache
/tmp/good/cache/sourcecred
/tmp/good/cache/sourcecred/example-github
/tmp/good/cache/sourcecred/example-github/github
/tmp/good/cache/sourcecred/example-github/git
/tmp/good/repositoryRegistry.json
/tmp/good/data
/tmp/good/data/sourcecred
/tmp/good/data/sourcecred/example-github
/tmp/good/data/sourcecred/example-github/github
/tmp/good/data/sourcecred/example-github/github/view.json
/tmp/good/data/sourcecred/example-github/git
/tmp/good/data/sourcecred/example-github/git/graph.json
```
wchargin-branch: load-pass-context
Adds a badge, adds a link, slight rewrite of the contributing section.
Test plan: Check the links and badge works.
(The badge topic count is off, we'll see if it is at least directionally
correct over time.)
Summary:
This improves the documentation for the change made in #715.
Test Plan:
Note that `./scripts/build_static_site.sh --help` and `yarn sharness`
pass.
wchargin-branch: feedback-url-help-text
Fixes#696.
Test plan: This is basically a config change, so I manually tested it.
I ran SourceCred on gitcoinco/web, which has two bots,
and verified that the bots are correctly removed from the list of users.
Selecting "Bots" in the dropdown filter shows the two bots. Changing
the user weight does not affect the bots' scores, and changing the bot
weight does affect the bots' scores.
Summary:
We can now set, at build time, a URL to be displayed at the top of the
prototype, encouraging users to provide feedback. If the URL is not
provided, it defaults to the appropriate topic on the SourceCred
Discourse instance.
The result looks like this:
![Screenshot of the feedback URL in the prototype][screenshot]
[screenshot]: https://user-images.githubusercontent.com/4317806/44814824-a238b380-ab92-11e8-88c8-dfbae27ca496.png
Test Plan:
Unit tests added to `yarn sharness-full` and `yarn unit`.
You can run `yarn start` to see the message with the default URL, or
`SOURCECRED_FEEDBACK_URL=http://example.com/ yarn start` to specify a
custom URL.
wchargin-branch: feedback-url
This commit adds a hardcoded list of known bots. Building on #713, it
categorizes those userlikes with the bot subtype. (Note that those users
may not be bots in the GitHub ontology - GitHub doesn't actually have a
clear record of which userlikes are bots.)
Progress towards #696.
Test plan:
Observe the single snapshot change, which demonstrates that @credbot is
now correctly categorized as a bot.
Summary:
As a first pass toward support for analyzing whole organizations, we
allow loading multiple repositories with `sourcecred load`, combining
them into a single relational view and a single Git graph at load time.
Test Plan:
Run
```
node bin/sourcecred.js \
load \
sourcecred/example-git \
sourcecred/example-github \
sourcecred/sourcecred \
--output sourcecred/examples \
;
```
and select `sourcecred/examples` from the web view. Filter “Repository”
nodes, and note that there are three.
Note that loading a single repository without `--output` still works,
that loading a single repository with `--output` still works (respecting
the alias name), and loading not exactly one repository without
`--output` yields an appropriate error message.
Note that `yarn sharness-full` still works.
wchargin-branch: load-combined
Userlikes now have an additional piece of data encoded in their address:
whether they are a USER or a BOT. Userlikes are still handled
identically by the RelationalView, which cuts down on code duplication.
I haven't added ORGANIZATIONs but it will be trivial to do once we're
interested in tracking them.
Note that this is basically the same as how we treat comments: comments
are subtyped to review comments, issue comments, and pull comments.
This is the initial step towards solving #696.
Test plan: Existing unit tests pass (and caught a few bugs during
development!). New test cases were added to the parser. Observe that all
the snapshot changes make sense.
Note: As of this commit, every GitHub userlike is classified as a user,
and the subtypes are not used in the application, so this commit causes
no change in observable behavior.
This commit changes the cred normalization algorithm so that the total
cred of all GitHub user nodes always sums to 1000. For rationale on the
change, see #705.
Fixes#705.
Note that this introduces a new way for PageRank to fail: if the
graph has no GitHub userlike nodes, then PageRank will throw an error
when it attempts to normalize. This will result in a message being
displayed to the user, and a more helpful error being printed to
console. If we need the cred explorer to display graphs that have no
userlike nodes, then we can modify the codepath so that it falls back to
normalizing based on all nodes instead of on the GitHub userlike nodes
specifically.
Test plan: There is an included unit test which verifies that the
new argument gets threaded through the state properly. But this is
mostly a config change, so it's best tested by actually inspecting
the cred explorer. I have done so, and can verify that the behavior is
as expected: the sum of users' cred now sums to 1000, and e.g. modifying
the weight on the repository node doesn't produce drastic changes to
cred scores.
This commit adds the logic for computing scores so that the total score,
summed across all nodes matching a NodePrefix, is a fixed constant.
See #705 for context.
Test plan: The logic is quite simple, and adequate unit tests are
included.
Note to reviewer: There is a spurious whitespace diff in the test file
because the tests for the previous test block were not correctly scoped.
Storing the user's weights in localStore enables a workflow where a
user chooses their preferred weights, and brings those weights with them
across projects and contexts. However, this is the wrong workflow:
actually, a project chooses its weights, and when a user visits a
particular project, they want to sync up with the project's choice.
Giving the user the ability to modify the weights and recalculate is
still important, so that they can propose improvements to the project
maintainer. But implicitly keeping their modified weights, and even
bringing them to other projects the user inspects, is
counter-productive.
This commit removes this dubious feature. (It's a feature we were likely
to drop anyway, as it conflicts with #703.) As an added bonus, this code
is untested, which means the feature is technical debt—so removing it
reduces our technical debt! It also removes at least one known bug.
Test plan: There are no tests. I manually verified that the frontend
still works, and that it no longer persists weights across refresh.
Summary:
By using Git’s magic pathspecs instead of post-processing stream
operations, we reduce the pipeline to a single operation. Git implements
its own version of `grep`, so this should be platform-independent.
Previously, we had needed the `-z` argument to `grep(1)`, which is a GNU
extension.
Fixes#594.
Test Plan:
Ensure that the script passes with no output. Then,
```shell
mkdir -p src/flow-typed/
touch src/foo.js src/flow-typed.js src/flow-typed/foo.js
git add src/foo.js src/flow-typed.js src/flow-typed/foo.js
cd scripts
./ensure-flow.js
```
and ensure that script exits with code 1, empty stdout, and stderr with:
```
../src/flow-typed.js
../src/flow-typed/foo.js
../src/foo.js
```
This verifies that the pathspec is properly excluding the root directory
`flow-typed`, but not files that just happen to have `flow-typed` in
their paths.
wchargin-branch: simplify-ensure-flow
Summary:
This patch adds independent exponential backoff to each individual
GitHub GraphQL query. We remove the fixed `GITHUB_DELAY_MS` delay before
each query in favor of this solution, which requires no additional
configuration (thus resolving a TODO in the process).
We use the NPM module `retry` with its default settings: namely, a
maximum of 10 retries with factor-2 backoff starting at 1000ms.
Empirically, it seems very unlikely that we should require much more
than 2 retries for a query. (See Test Plan for more details.)
This is both a short-term unblocker and a good kind of thing to have in
the long term.
Test Plan:
Note that `yarn test --full` passes, including `fetchGithubRepoTest.sh`.
Consider manual testing as follows.
Add `console.info` statements in `retryGithubFetch`, then load a large
repository like TensorFlow, and observe the output:
```shell
$ node bin/sourcecred.js load --plugin github tensorflow/tensorflow 2>&1 | ts -s '%.s'
0.252566 Fetching repo...
0.258422 Trying...
5.203014 Trying...
[snip]
1244.521197 Trying...
1254.848044 Will retry (n=1)...
1260.893334 Trying...
1271.547368 Trying...
1282.094735 Will retry (n=1)...
1283.349192 Will retry (n=2)...
1289.188728 Trying...
[snip]
1741.026869 Ensuring no more pages...
1742.139978 Creating view...
1752.023697 Stringifying...
1754.697116 Writing...
1754.697772 Done.
```
This took just under half an hour, with 264 queries total, of which:
- 225 queries required 0 retries;
- 38 queries required exactly 1 retry;
- 1 query required exactly 2 retries; and
- 0 queries required 3 or more retries.
wchargin-branch: github-backoff
Summary:
The current version of the build script has the safe but annoying
property that the target directory must be an existing, empty directory.
It seems reasonable and convenient to allow the build script to create
the directory with `mkdir -p`. It still fails if the directory is not
empty or is a file.
Test Plan:
Unit tests updated; run `yarn sharness-full`.
wchargin-branch: build-mkdir-p
Summary:
The version number displayed in the application now displays much more
specific information. It now lists the Git commit from which the build
was constructed, and will identify whether we have accidentally deployed
a development instance (which would be slow) or an instance with
uncommitted changes (which would be bad).
The version information is computed during the initialization of the
Webpack config. For development, this means that it is computed when you
run `yarn start`, and not updated thenafter. If the stale information
presents actual confusion, we would need to backport Webpack 4’s support
for runtime values in `DefinePlugin` to Webpack 3 (or upgrade Webpack
by a major version).
Test Plan:
The logic for `GitState` and `Environment` has existing tests. With both
a clean tree and a dirty tree, run `yarn start` and build the static
site, and check that the resulting versions are correct.
wchargin-branch: use-rich-version-types
Summary:
These types will shortly be added to the global `VersionInfo`. For now,
we include the types and validation logic only.
Test Plan:
Unit tests suffice.
wchargin-branch: add-rich-version-types
This commit re-introduces the git plugin, now that it has been radically
simplified as described in [1]. The new git plugin only has nodes for
commits and only has commit has-parent edges. As compared to the version
that was removed in #628, this plugin is far leaner. It doesn't bloat
the graph (for `sourcecred/sourcecred`, the git plugin data is just
164k), and as such doesn't incur much performance penalty.
Re-incorporating the git plugin also brings some tangible benefits. We
already had git nodes in the graph, as the GitHub plugin attaches them
to pull requests. Without any git plugin, these nodes are displayed as
"uknown nodes" with ugly descriptions. Also, including a git plugin,
even one that is very minimal, communicates to users that git is a
source of information to SourceCred, and that they can expect more from
it in the future.
Note that this commit breaks backcompat for existing repositories that
were locally loaded after #628. As such, it is best to
`rm -rf $SOURCECRED_DIRECTORY` and start with fresh data. Also, due to a
known bug in the WeightConfig, you should reset your browser's local
storage.
Test plan: After removing the SourceCred directory and the stale
localStorage, the cred explorer nicely displays git commits, and
connects them via has_parent edges. The NodeType filter allows filtering
to commits as expected, and the WeightConfig shows node and edge weights
for the Git plugin's nodes and edges.
[1]: https://github.com/sourcecred/sourcecred/issues/627#issuecomment-413435447
The minimal git plugin adapter only provides commit nodes and has_parent
edges. See #627 for context.
I forked this from `git/pluginAdapter.js`, and then deleted the
nodeTypes and edgeTypes which are no longer in scope.
Test plan: This is a fork of untested "glue" code, and is itself still
untested.
This implements the approach suggested in [1]. Instead of forking the
git plugin entirely, we'll fork the createGraph method and the
pluginAdapter so that we have instances that produce a lightweight git
graph.
createMinimalGraph is a fork of createGraph that only adds commit nodes
and has_parent edges. New unit tests ensure that only the whitelisted
nodes and edges appear.
Supersedes #683 and #684.
Test plan: `yarn test`
[1]: https://github.com/sourcecred/sourcecred/issues/627#issuecomment-413623784
Summary:
We don’t use or want these. Injecting an arbitrary family of variables
from the client’s host environment seems like a Bad Idea.
Test Plan:
The usual `yarn start`, static site, and `yarn test --full` still work.
wchargin-branch: remove-reactapp-vars
Summary:
Cargo-culting `reduce` doesn’t make something “functional” or “good”;
forcing a `for`-loop into a `reduce` with impure callback is abhorrent.
Test Plan:
Into `config/stopship.js`, write:
```js
console.log(JSON.stringify(require("./env")()));
```
Then run `NODE_ENV=test node config/stopship.js` before and after this
commit and note that the output is identical.
wchargin-branch: simplify-getClientEnvironment
Summary:
Now that the main functionality of #643 has been implemented, we no
longer have any use for the “public URL” property. In fact, its presence
is actively harmful, as it suggests that the gateway may be known before
runtime, which is confusing and false.
Closes#643.
Test Plan:
Running `yarn start` works. Building the static site works.
Invoking `git grep -i 'public.\?url'` finds no matches.
Also, `yarn test --full` passes.
wchargin-branch: remove-public-url
We often construct case statements over union-typed variables, and then
in the default case, we use a `(type: empty)` assertion to ensure that
failing to account for all the cases results in a flow error.
In the past, we created an extra line for this assertion, which required
some eslint suppressions. We've realized it's cleaner to inline the type
assertion in the runtime error that we throw in these defaults.
This code cleans everything to the new style, and removes every existing
`// no-unused-expressions` invocation in the codebase.
Test plan: `yarn test`
It's more consistent to prepend entries to the [Unreleased] section of
the changelog, so that entries are all in reverse-chronological order.
Since we've appended the first few entries, we reverse them now.
Test plan: Not needed
The 'Score' column is renamed to 'Cred' (and its prop is renamed as
well). The column which shows how a connection or aggregation
contributes to a node's cred, as a percentage, has been rendered
nameless. It is pretty self explanatory, and the previous name
("Connection") was meaningless.
Test plan: Unit tests, also I inspected the frontend.
Some CSS magic was required.
Also creates `src/app/version.js` for storing the version string.
Test plan: Visual inspection of the footer in both Chrome and Firefox,
both on a page with very little content (the cred explorer without a
repository loaded), and on a page with more than a screen height's of
content (the homepage, or cred explorer with a large repository loaded).
In all cases, the footer unobtrusively appears in the lower-left hand
corner at the bottom of the screen, (after scrolling past all content,
if applicable).
Summary:
The initial logo checkin in #637 included the 32px raster image, but
generated it in the _wrong order_ in the rasterizer script. This commit
fixes that heinous bug once and for all.
Test Plan:
Running `rasterize.sh` does not change the output.
wchargin-branch: rasterize-32px
Summary:
This commit approximately completes the implementation of #643.\* Plugin
adapters are now provided an `Assets` object at `load` time, which they
can use to resolve their plugin-specific API routes.
\* “Approximately” because there are some non-essential pieces of legacy
code that should be cleaned up.
Test Plan:
Unit tests modified, but it would be good to also manually test this.
Run `./scripts/build_static_site.sh` to build the site to, say,
`/tmp/gateway/`. Then spin up a static HTTP server serving `/tmp/` and
navigate to `/gateway/` in the browser. Note that the entire application
works.
wchargin-branch: use-assets-in-PluginAdapters
Summary:
This commit is the next step in #643. It makes the `RepositorySelect`
robust to being hosted at arbitrary gateways by accepting `Assets` and
resolving the repository registry API route appropriately.
Test Plan:
Unit tests modified, but it would be good to also manually test this.
Run `./scripts/build_static_site.sh` to build the site to, say,
`/tmp/gateway/`. Then spin up a static HTTP server serving `/tmp/` and
navigate to `/gateway/` in the browser. Note that you can navigate
around the application and load the repository registry on the prototype
without any console warnings or errors, although you cannot yet load
actual graph data.
wchargin-branch: use-assets-in-RepositorySelect
Summary:
This commit takes the next step toward #643 by exposing `Assets` to our
React components at top level. Components will be expected to pass them
down as appropriate; this commit does not add any actual uses.
Test Plan:
Apply the following patch:
```diff
diff --git a/src/app/Page.js b/src/app/Page.js
index 24c2602..7ac2641 100644
--- a/src/app/Page.js
+++ b/src/app/Page.js
@@ -24,6 +24,10 @@ export default class Page extends React.Component<{|
<Link to="/" className={css(style.navLink, style.navLinkTitle)}>
SourceCred
</Link>
+ <img
+ alt="fav"
+ src={this.props.assets.resolve("/favicon.png")}
+ />
</li>
{routeData.map(({navTitle, path}) =>
NullUtil.map(navTitle, (navTitle) => (
```
Then, observe that the favicon loads correctly and updates across page
loads and refreshes in the following situations:
- under `yarn start`;
- after building the static site and serving from root;
- after building the static site and serving from another gateway.
wchargin-branch: use-withAssets
Summary:
This is the last piece of major infrastructure for #643. It will enable
components like `Page` and `CredExplorerApp` to receive `Assets` as a
prop.
A previous iteration of the same functionality used the new Context API
in React v16.3. This did a good job of solving the problem in production
code, and was convenient. However, it is currently intractable to test
with the current state of Enzyme. It’s plausible that this might improve
in the future, so if threading down the props becomes to onerous, we
might check in to see how our testing libraries are doing. I expect that
the threading should not be too bad, given that we do the same thing
with `localStore`, which has worked (as far as I’m aware) without a
hitch.
Test Plan:
Unit tests added; `yarn test` suffices.
wchargin-branch: withAssets
Summary:
This is copied from the Sharness repository’s `test/.gitignore`, and is
also the same as Git’s `t/.gitignore` minus a Git-specific exclusion.
Suggested by @decentralion.
Test Plan:
Run `yarn sharness` and interrupt it (SIGINT) while `make` is running.
Note that `sharness/` now contains a `trash directory.*`, which Git
ignores.
wchargin-branch: sharness-gitignore
Summary:
As the next step for #643, this patch enables the app to be rendered at
non-root gateways by incorporating the relative-path history
implementation developed in #666. The app is not fully functional:
our React components do not yet know how to resolve assets, and so
fetches of resources like the repository will be against the wrong URLs.
Test Plan:
- Note that `yarn start` still works.
- Run `./scripts/build_static_site.sh` to build the site into, say,
`/tmp/gateway`.
- Run a static web server from `/tmp/gateway/` and note that (a) the
paths listed in the page source are relative, and (b) everything
works as intended, with no console messages in either Chrome or
Firefox.
- Run a static web server from `/tmp/` and navigate to `/gateway/` in
the browser. Note that the app loads properly, and that refreshes
work (i.e., the `pushState` paths are real paths). Note that the
repository registry cannot yet be loaded, and so PageRank cannot be
run.
wchargin-branch: relative-router
Summary:
This is the first observable step toward #643. Assets whose paths are
known as literals at server-side rendering time are now referenced via
relative paths. This means that the favicon and JavaScript bundle can be
loaded from an arbitrary gateway. The actual bundle code will still only
work when loaded from `/`.
This commit stands alone so that the enclosing change to the Webpack
config can be in as small a change as possible.
Test Plan:
- Note that `yarn start` still works.
- Run `./scripts/build_static_site.sh` to build the site into, say,
`/tmp/gateway`.
- Run a static web server from `/tmp/gateway/` and note that (a) the
paths listed in the page source are relative, and (b) everything
works as intended, with no console messages in either Chrome or
Firefox.
- Run a static web server from `/tmp/` and navigate to `/gateway/` in
the browser. Note that the favicon and JavaScript are correctly
noted, but that the router raises an error because it is trying to
load a non-existent route. (This behavior is unchanged.)
wchargin-branch: relative-lexically-static
Previously, expanding a node would display the individual connections
that contributed cred to that node. For nodes with high degree, this was
a pretty noisy UI.
Now, expanding a node displays "aggregations": for every type of
adjacent connection (where type is the union of the edge type and the
adjacent node type), we show a summary of the total cred from
connections of that type. The result is a much more managable summary
view. Naturally, these aggregations can be further expanded to see the
individual connections.
Closes#502.
Test plan: The new behavior is unit tested. You can also launch the cred
explorer and experience the UI directly. I have used the new UI a lot,
as well as demo'd it to people, and I like it quite a bit.
For the group of aggregations returned by aggregation operation (e.g.
the set of aggregations returned by a call to `flatAggregate`), the keys
are unique.
Test plan: `yarn test`
The TableRow currently has some margin on the left, but not on the
right. This is visually unbalanced, especially when expanded so depth>0,
as the content on the right is at the very edge of the shaded rectangle.
This commit cleans that up a bit!
Test plan: Visual inspection (see screenshots in the pull request). I
don't think unit tests are necessary for small visual tweaks like this.
Summary:
This is necessary for #643. If we’re serving `/prototype/index.html`, we
need to to use `..` to refer to the root of the site. This patch adds
`rootFromPath`, which performs the relevant transformation. (The
implementation is trivial, but figuring out exactly what the
specification should be was not!)
Test Plan:
Unit tests added; `yarn test` suffices.
wchargin-branch: rootFromPath
Summary:
This will enable clients to obtain the path to a static asset, even when
the app is not hosted at the root of a server, as outlined in #643.
This module will be used for simple assets (images, etc.) and API data
(fetches from `/api/**`) alike.
This supersedes #663. It includes the logic from that PR (`Assets`)
without the React-specific context bindings (`AssetsContext`).
Test Plan:
Unit tests included; `yarn test` suffices.
wchargin-branch: assets-resolver
Summary:
See #643 and the module docstring on `createRelativeHistory.js` for
context and explanation.
This patch adds `history@^3.0.0` as an explicit dependency—previously,
we were depending on it only implicitly through `react-router` (which
was fine then, but is not now). The dependency is chosen to match the
version specified in `react-router`’s `package.json`.
Test Plan:
Extensive unit tests included, with full coverage; `yarn test` suffices.
wchargin-branch: createRelativeHistory
William Chargin
Dandelion Mané
