sourcecred/CHANGELOG.md

# Changelog

## [Unreleased]
- Hyperlink to GitHub entities (#860)
- Add GitHub reactions to the graph (#846)
- Detect references to commits (#833)
- Detect references in commit messages (#829)
- Add commit authorship to the graph (#826)
- Add `MentionsAuthor` edges to the graph (#808)
<!-- Please add new entries to the _top_ of this section. -->

## [0.1.0]
- Organize weight config by plugin (#773)
- Configure edge forward/backward weights separately (#749)
- Combine "load graph" and "run pagerank" into one button (#759)
- Store GitHub data compressed at rest, reducing space usage by 6–8× (#750)
- Improve weight sliders display (#736)
- Separate bots from users in the UI (#720)
- Add a feedback link to the prototype (#715)
- Support combining multiple repositories into a single graph (#711)
- Normalize scores so that 1000 cred is split amongst users (#709)
- Stop persisting weights in local store (#706)
- Execute GraphQL queries with exponential backoff (#699)
- Introduce a simplified Git plugin that only tracks commits (#685)
- Rename cred explorer table columns (#680)
- Display version string in the app's footer
- Support hosting SourceCred instances at arbitrary gateways, not just
  the root of a domain (#643)
- Aggregate over connection types in the cred explorer (#502)
- Start tracking changes in `CHANGELOG.md`
-												Add CHANGELOG.md (#670)

Also, update CONTRIBUTING.md to guide contributors to update the
changelog.

Test plan: Unnecessary
											
										
										
											2018-08-15 22:20:59 +00:00
+								# Changelog
 								## [Unreleased]
-												Display urls in the cred explorer (#860)

This commit modifies the plugin adapter's `nodeDescription` method so
that it may return a React node.

This enables the GitHub plugin's `nodeDescription` method to include
hyperlinks directly to the referenced content on GitHub. This makes
examining e.g. comment cred much easier.

I've also made two other changes to the descriptions:
- Pull requests diffs now color-encode the additions and deletions
- Descriptions for comments and reviews no longer include the authors

The Git plugin's behavior is unchanged.

Test plan:
I loaded a large repository in the cred explorer and verified that
exploring comments and pulls and issues is much easier. The descriptions
are as expected for every category of node. Snapshot tests updated.

Fixes #590.
											
										
										
											2018-09-20 17:48:05 +00:00
+								- Hyperlink to GitHub entities (#860)
-												Add GitHub reactions to the graph (#846)

* Define Reaction edges

This adds support to `github/edges` for creating edges representing
GitHub reactions. These edges are not actually added to the graph.

Test plan: Unit tests

* Add GitHub reactions to the graph

This commit adds functional support for reactions in SourceCred.
Only thumbs-up, heart, and hooray reactions are supported for now, as
they are all unambiguously positive; adding support for negative
reactions like thumbs-down will require some more thought.

The reactions are added to the graph, and new edge types have been added
to the UI.

Test plan:
The `graphView` class has been updated to do invariant checking for the
reaction edges, including that the unsupported reaction types like
"THUMBS_DOWN" aren't added to the graph.

I've tested this feature by downloading data for a large repository
(ipfs/go-ipfs). The reaction edges appear and transfer cred reasonably.
The edge types are displayed in the weight config appropriately.

Builds on #839, #840, and #845.

											
										
										
											2018-09-17 20:44:11 +00:00
+								- Add GitHub reactions to the graph (#846)
-												Detect references to commits (#833)

Now that #832 gave us logic to parse references to commits, we have the
RelationalView find and add these references. The actual change is
a simple extension of existing reference detection logic.

Test plan: Observe that the snapshots are updated with references to
commits from the example-github repository.

Progress on #815.
											
										
										
											2018-09-14 18:56:16 +00:00
+								- Detect references to commits (#833)
-												Detect references in commit messages (#829)

Now that the GitHub plugin knows about commit messages (#828), we can
parse those commit messages to find references to other GitHub entities.

Fixed a minor typing mistake along the way.

Test plan:
Observe that a number of references have been detected among the commits
in the example GitHub repository. We mistakenly find references to
wchargin because we don't have a proper tokenizer. (#481)

Progress on #815.
											
										
										
											2018-09-13 22:46:39 +00:00
+								- Detect references in commit messages (#829)
-												Add commit authorship to the graph (#826)

In #824, we loaded every commit in the default branch's history into the
GitHub relational view, along with authorship info. This commit actually
uses that authorship info to create AUTHORS edges from the commit to the
user that authored it (whenever possible).

The implementation is quite simple: we just need to yield the commits
when we yield all the authored entities, so that we will process their
authors and add them to the graph. Also, I updated the invariant
declarations in `graphView.js`, and corrected a type signature so that the
new invariants would typecheck.

Test plan: The snapshot update shows that commits are being added to the
graph appropriately. Observe that commits which do not have a valid
GitHub user as their author do not correspond to edges in the graph.
See [example].

This is basically a solution to #815, but I'll defer closing that issue
until I've added a few more features, like reference detection.

[example]: https://github.com/sourcecred/example-github/commit/6bd1b4c0b719c22c688a74863be07a699b7b9b34
											
										
										
											2018-09-13 21:19:37 +00:00
+								- Add commit authorship to the graph (#826)
-												Update CHANGELOG.md (#820)

It now mentions that we added `MentionsAuthor` edges to the GitHub
graph in #808.

Thanks @whyrusleeping for suggesting this heuristic.

Test plan: n/a
											
										
										
											2018-09-13 03:30:35 +00:00
+								- Add `MentionsAuthor` edges to the graph (#808)
-												Release 0.1.0 (#799)

Test plan: I ran `yarn test --full`. I also regenerated data from
scratch and manually tested the cred explorer.
											
										
										
											2018-09-07 02:06:16 +00:00
+								<!-- Please add new entries to the _top_ of this section. -->
 								## [0.1.0]
-												Organize weights by plugin (#773)

This commit adds PluginWeightConfig, which is responsible for
adding all the weights for an individual plugin. The top-level
WeightConfig now creates multiple PluginWeightConfigs. It also takes
responsibility for hiding the FallbackPlugin.

Test plan: The PluginWeightConfig is tested (and fairly simple). The
top-level WeightConfig is not yet tested (#604), so I manually tested
that the weights in the app still function.
											
										
										
											2018-09-05 18:57:20 +00:00
+								- Organize weight config by plugin (#773)
-												Configure forward/backward edge weights separately (#749)

This commit introduces a new component, `EdgeTypeConfig`, which is
responsible for configuring the weights for a given edge type. The
config creates two `WeightSlider`s: one for the forward direction, and
one for the backward direction. The `DirectionalitySlider` is no longer
used, and is removed. This fixes #596.

So as to avoid confusion, we now describe every edge with variables, as
in 'α REFERENCES β', and clarify that the weight modifies how cred flows
from β to α. This necessitated the creation of an `EdgeWeightSlider`,
local to the `EdgeTypeConfig`, which sets up a `WeightSlider` with the
necessary greek characters.

The EdgeTypeConfig is tested, so this is continuing progress towards
solving #604.

Test plan: I manually verified that modifying edge weights has the
expected effect on cred scores. Also, some new unit tests are included.
											
										
										
											2018-09-04 22:37:00 +00:00
+								- Configure edge forward/backward weights separately (#749)
-												Combine loadGraph and runPagerank into one button (#759)

* StateTransitionMachine.loadGraph reports success

Step one towards #586. This will enable us to chain runPagerank after
loadGraph only if the load went through successfully.

Test plan: Unit tests included.

* Add StateTransitionMachine.loadGraphAndRunPagerank

This methods combines `loadGraph` and `runPagerank` into one method
which internally chains the two method. `runPagerank` is only called if
`loadGraph` was successful.

Progress on #586.

Test plan:
The new method has attached unit tests. I implemented the unit tests via
mocking, which seemed quite convenient as the method is basically a
wrapper for chaining two other function calls.

* Combine loadGraph and runPagerank into one button

Resolves #586. The new button is called "Analyze cred".

Test plan: Unit tests, also I tested it manually.

											
										
										
											2018-09-03 21:34:14 +00:00
+								- Combine "load graph" and "run pagerank" into one button (#759)
-												Store GitHub data gzipped at rest (#751)

Summary:
We store the relational view in `view.json.gz` instead of `view.json`,
taking advantage of the isomorphic `pako` library for gzip encoding and
decoding.

Sample space savings (note that post bodies are included; i.e., #747 has
not been applied):

       SAVE     OLD (B)     NEW (B) REPO
      89.7%       25326        2617 sourcecred/example-github
      82.9%     3257576      555948 sourcecred/sourcecred
      85.2%    11287621     1665884 ipfs/js-ipfs
      88.0%    20953425     2520358 gitcoinco/web
      84.4%    38196825     5951459 ipfs/go-ipfs
      84.9%   205770642    31101452 tensorflow/tensorflow

<details>
<summary>Script to generate space savings output</summary>

```shell
savings() {
    printf '% 7s % 11s % 11s %s\n' 'SAVE' 'OLD (B)' 'NEW (B)' 'REPO'
    for repo; do
        file="${SOURCECRED_DIRECTORY}/data/${repo}/github/view.json.gz"
        if ! [ -f "${file}" ]; then
            printf >&2 'warn: no such file %s\n' "${file}"
            continue
        fi
        script="$(sed -e 's/^ *//' <<EOF
            repo = '${repo}'
            pre_size = $(<"${file}" gzip -dc | wc -c)
            post_size = $(<"${file}" wc -c)
            percentage = '%0.1f%%' % (100 * (1 - post_size / pre_size))
            p = '% 7s % 11d % 11d %s' % (percentage, pre_size, post_size, repo)
            print(p)
EOF
        )"
        python3 -c "${script}"
    done
}
```

</details>

Closes #750.

Test Plan:
Comparing the raw old version with the decompressed new version shows
that they are identical:

```
$ <~/tmp/sourcecred/data/sourcecred/example-github/github/view.json \
> shasum -a 256 -
63853b9d3f918274aafacf5198787e18185a61b9c95faf640a1e61f5d11fa19f  -
$ <~/tmp/sourcecred/data/sourcecred/example-github/github/view.json.gz \
> gzip -dc | shasum -a 256
63853b9d3f918274aafacf5198787e18185a61b9c95faf640a1e61f5d11fa19f  -
```

Additionally, `yarn test --full` passes, and `yarn start` still loads
data and runs PageRank properly.

wchargin-branch: gzip-relational-view

											
										
										
											2018-09-01 17:42:30 +00:00
+								- Store GitHub data compressed at rest, reducing space usage by 6–8× (#750)
-												Better handling of log weights (#736)

This commit isolates all of the log-weight behavior in the weight
slider. That slider moves in log space, but the numbers printed and
passed around the WeightConfig code are now always in linear-space.

This should reduce confusion in the UI and for developers.

This commit contains two other improvements: (#588)
- Changes the (log space) range on the sliders from ±10 to ±5
- Change the order from slider, weight, name to name, slider, weight, so
that there is more visual separation between the name and the weight.

Test plan: Changes to the weight slider are tested. Changes to the
WeightConfig aren't (#604) so I manually tested the UI.
											
										
										
											2018-08-31 02:21:59 +00:00
+								- Improve weight sliders display (#736)
-												Separate bots and users in the UI (#720)

Fixes #696.

Test plan: This is basically a config change, so I manually tested it.
I ran SourceCred on gitcoinco/web, which has two bots,
and verified that the bots are correctly removed from the list of users.
Selecting "Bots" in the dropdown filter shows the two bots. Changing
the user weight does not affect the bots' scores, and changing the bot
weight does affect the bots' scores.
											
										
										
											2018-08-29 22:14:42 +00:00
+								- Separate bots from users in the UI (#720)
-												Add a configurable feedback URL to prototype (#715)

Summary:
We can now set, at build time, a URL to be displayed at the top of the
prototype, encouraging users to provide feedback. If the URL is not
provided, it defaults to the appropriate topic on the SourceCred
Discourse instance.

The result looks like this:

![Screenshot of the feedback URL in the prototype][screenshot]

[screenshot]: https://user-images.githubusercontent.com/4317806/44814824-a238b380-ab92-11e8-88c8-dfbae27ca496.png

Test Plan:
Unit tests added to `yarn sharness-full` and `yarn unit`.

You can run `yarn start` to see the message with the default URL, or
`SOURCECRED_FEEDBACK_URL=http://example.com/ yarn start` to specify a
custom URL.

wchargin-branch: feedback-url
											
										
										
											2018-08-29 22:06:12 +00:00
+								- Add a feedback link to the prototype (#715)
-												Allow combining repositories at load time (#711)

Summary:
As a first pass toward support for analyzing whole organizations, we
allow loading multiple repositories with `sourcecred load`, combining
them into a single relational view and a single Git graph at load time.

Test Plan:
Run

```
node bin/sourcecred.js \
    load \
    sourcecred/example-git \
    sourcecred/example-github \
    sourcecred/sourcecred \
    --output sourcecred/examples \
    ;
```

and select `sourcecred/examples` from the web view. Filter “Repository”
nodes, and note that there are three.

Note that loading a single repository without `--output` still works,
that loading a single repository with `--output` still works (respecting
the alias name), and loading not exactly one repository without
`--output` yields an appropriate error message.

Note that `yarn sharness-full` still works.

wchargin-branch: load-combined
											
										
										
											2018-08-29 21:52:26 +00:00
+								- Support combining multiple repositories into a single graph (#711)
-												Users have 1000 cred in aggregate (#709)

This commit changes the cred normalization algorithm so that the total
cred of all GitHub user nodes always sums to 1000. For rationale on the
change, see #705.

Fixes #705.

Note that this introduces a new way for PageRank to fail: if the
graph has no GitHub userlike nodes, then PageRank will throw an error
when it attempts to normalize. This will result in a message being
displayed to the user, and a more helpful error being printed to
console. If we need the cred explorer to display graphs that have no
userlike nodes, then we can modify the codepath so that it falls back to
normalizing based on all nodes instead of on the GitHub userlike nodes
specifically.

Test plan: There is an included unit test which verifies that the
new argument gets threaded through the state properly. But this is
mostly a config change, so it's best tested by actually inspecting
the cred explorer. I have done so, and can verify that the behavior is
as expected: the sum of users' cred now sums to 1000, and e.g. modifying
the weight on the repository node doesn't produce drastic changes to
cred scores.
											
										
										
											2018-08-29 19:20:57 +00:00
+								- Normalize scores so that 1000 cred is split amongst users (#709)
-												Stop persisting users' weight choices (#706)

Storing the user's weights in localStore enables a workflow where a
user chooses their preferred weights, and brings those weights with them
across projects and contexts. However, this is the wrong workflow:
actually, a project chooses its weights, and when a user visits a
particular project, they want to sync up with the project's choice.
Giving the user the ability to modify the weights and recalculate is
still important, so that they can propose improvements to the project
maintainer. But implicitly keeping their modified weights, and even
bringing them to other projects the user inspects, is
counter-productive.

This commit removes this dubious feature. (It's a feature we were likely
to drop anyway, as it conflicts with #703.) As an added bonus, this code
is untested, which means the feature is technical debt—so removing it
reduces our technical debt! It also removes at least one known bug.

Test plan: There are no tests. I manually verified that the frontend
still works, and that it no longer persists weights across refresh.
											
										
										
											2018-08-29 18:46:48 +00:00
+								- Stop persisting weights in local store (#706)
-												Retry GitHub queries with exponential backoff (#699)

Summary:
This patch adds independent exponential backoff to each individual
GitHub GraphQL query. We remove the fixed `GITHUB_DELAY_MS` delay before
each query in favor of this solution, which requires no additional
configuration (thus resolving a TODO in the process).

We use the NPM module `retry` with its default settings: namely, a
maximum of 10 retries with factor-2 backoff starting at 1000ms.
Empirically, it seems very unlikely that we should require much more
than 2 retries for a query. (See Test Plan for more details.)

This is both a short-term unblocker and a good kind of thing to have in
the long term.

Test Plan:
Note that `yarn test --full` passes, including `fetchGithubRepoTest.sh`.
Consider manual testing as follows.

Add `console.info` statements in `retryGithubFetch`, then load a large
repository like TensorFlow, and observe the output:

```shell
$ node bin/sourcecred.js load --plugin github tensorflow/tensorflow 2>&1 | ts -s '%.s'
0.252566 Fetching repo...
0.258422 Trying...
5.203014 Trying...
[snip]
1244.521197 Trying...
1254.848044 Will retry (n=1)...
1260.893334 Trying...
1271.547368 Trying...
1282.094735 Will retry (n=1)...
1283.349192 Will retry (n=2)...
1289.188728 Trying...
[snip]
1741.026869 Ensuring no more pages...
1742.139978 Creating view...
1752.023697 Stringifying...
1754.697116 Writing...
1754.697772 Done.
```

This took just under half an hour, with 264 queries total, of which:
  - 225 queries required 0 retries;
  - 38 queries required exactly 1 retry;
  - 1 query required exactly 2 retries; and
  - 0 queries required 3 or more retries.

wchargin-branch: github-backoff
											
										
										
											2018-08-22 18:37:29 +00:00
+								- Execute GraphQL queries with exponential backoff (#699)
-												Re-introduce a simplified git plugin (#685)

This commit re-introduces the git plugin, now that it has been radically
simplified as described in [1]. The new git plugin only has nodes for
commits and only has commit has-parent edges. As compared to the version
that was removed in #628, this plugin is far leaner. It doesn't bloat
the graph (for `sourcecred/sourcecred`, the git plugin data is just
164k), and as such doesn't incur much performance penalty.

Re-incorporating the git plugin also brings some tangible benefits. We
already had git nodes in the graph, as the GitHub plugin attaches them
to pull requests. Without any git plugin, these nodes are displayed as
"uknown nodes" with ugly descriptions. Also, including a git plugin,
even one that is very minimal, communicates to users that git is a
source of information to SourceCred, and that they can expect more from
it in the future.

Note that this commit breaks backcompat for existing repositories that
were locally loaded after #628. As such, it is best to
`rm -rf $SOURCECRED_DIRECTORY` and start with fresh data. Also, due to a
known bug in the WeightConfig, you should reset your browser's local
storage.

Test plan: After removing the SourceCred directory and the stale
localStorage, the cred explorer nicely displays git commits, and
connects them via has_parent edges. The NodeType filter allows filtering
to commits as expected, and the WeightConfig shows node and edge weights
for the Git plugin's nodes and edges.

[1]: https://github.com/sourcecred/sourcecred/issues/627#issuecomment-413435447
											
										
										
											2018-08-16 20:20:41 +00:00
+								- Introduce a simplified Git plugin that only tracks commits (#685)
-												Rename cred explorer table columns (#680)

The 'Score' column is renamed to 'Cred' (and its prop is renamed as
well). The column which shows how a connection or aggregation
contributes to a node's cred, as a percentage, has been rendered
nameless. It is pretty self explanatory, and the previous name
("Connection") was meaningless.

Test plan: Unit tests, also I inspected the frontend.
											
										
										
											2018-08-16 05:22:21 +00:00
+								- Rename cred explorer table columns (#680)
-												Reverse the order of CHANGELOG entries (#681)

It's more consistent to prepend entries to the [Unreleased] section of
the changelog, so that entries are all in reverse-chronological order.
Since we've appended the first few entries, we reverse them now.

Test plan: Not needed
											
										
										
											2018-08-16 18:14:52 +00:00
+								- Display version string in the app's footer
 								- Support hosting SourceCred instances at arbitrary gateways, not just
 								  the root of a domain (#643)
 								- Aggregate over connection types in the cred explorer (#502)
 								- Start tracking changes in `CHANGELOG.md`
-												Add CHANGELOG.md (#670)

Also, update CONTRIBUTING.md to guide contributors to update the
changelog.

Test plan: Unnecessary
											
										
										
											2018-08-15 22:20:59 +00:00