sourcecred/CHANGELOG.md

50 lines
2.1 KiB
Markdown
Raw Normal View History

# Changelog
## [Unreleased]
Add CLI command to clear sourcecred data directory (#1111) Resolves #1067 Adds the CLI commands: `sourcecred clear --all` -- removes the $SOURCECRED_DIRECTORY `sourcecred clear --cache` -- removes the cache directory `sourcecred clear --help` -- provides usage info `sourcecred clear` -- prompts the user to be more specific Test plan: The unit tests ensure that the command is properly wired into the sourcecred CLI, including help text integration. However, just to be safe, we can start by verifying that calling `sourcecred` without arguments lists the `clear` command as a valid option, and that calling `sourcecred help clear` prints help information. (Note: it's necessary to run `yarn backend` before testing these changes) The unit tests also ensure that the command removes the proper directories, so there isn't really a need to manually test it, although the reviewer may choose to do so to be safe. Although out of scope for unit tests on this function, we can also do integration tests, to make sure that running the clear command doesn't leave the sourcecred directory in an invalid state from the perspective of the `load` command. ```js $ yarn backend; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --cache; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --all; $ node bin/sourcecred.js load sourcecred/example-github; ``` The expected behavior of the above command block is that the load command never fails or throws an error. @decentralion and I discussed the scenario where `rimraf` errors. We decided that testing this scenario wasn't necessary, because `rimraf` doesn't error if a directory doesn't exist, and rimraf's maintainer suggests [monkey-patching the fs module] to get rimraf to error in testing scenarios. Thanks @decentralion for reviewing and pair-programming this with me. [monkey-patching the fs module]: https://github.com/isaacs/rimraf/issues/31#issuecomment-29534796
2019-05-13 09:59:58 +00:00
- Allow the user to save or upload weight settings (#1150)
explorer: tweak weights on a per-node basis (#1143) This pull request adds a weight slider to every NodeRow in the explorer, enabling the user to manually set a weight for that node. The weights are multiplicative with the type level weights, so that they can be changed independently (e.g. you can have a comment that is weighted 2x higher than regular comments, but still have comments get a low weight in general). This pull coordinates a number of different changes across the codebase, all of which are tested: Adding support for manual weights in the weights and weightsToEdgeEvaluator modules. Modifying pagerankTable.TableRow so that it can show a slider in the second column. Adding piping for manual weights into the PagerankTable shared props, and into the explorer app Adding the slider to the NodeRow class that displays the current weight, and can trigger the upstream weight change Ensuring that the runPagerank call in the explorer actually uses the manual weights At present, there is no way to save these weights (they are ephemeral in the frontend) and so this is clearly a prototype/tech demo level feature rather than being ready for real usage. Correspondingly, CLI pagerank command always uses an empty set of manual weights. I plan to remedy this in a follow-on pull request. Test plan: Run the included unit tests (yarn test) and also spin up the UI, verify that it visually looks good in both Firefox and Chrome, and verify that changing the weights and then re-running PageRank actually causes the cred of the modified node to change. Review plan: In addition to carefully reading the code, ensure that all of the changes described a few paragraphs up are actually tested. Merge plan: Squash and merge. Thanks to @s-ben for proposing this feature in Discord, and to everyone discussing its implications in this Discourse thread.
2019-05-18 16:20:27 +00:00
- Allow tweaking weights on a per-node basis (#1143)
- Add the `pagerank` command (#1114)
Add CLI command to clear sourcecred data directory (#1111) Resolves #1067 Adds the CLI commands: `sourcecred clear --all` -- removes the $SOURCECRED_DIRECTORY `sourcecred clear --cache` -- removes the cache directory `sourcecred clear --help` -- provides usage info `sourcecred clear` -- prompts the user to be more specific Test plan: The unit tests ensure that the command is properly wired into the sourcecred CLI, including help text integration. However, just to be safe, we can start by verifying that calling `sourcecred` without arguments lists the `clear` command as a valid option, and that calling `sourcecred help clear` prints help information. (Note: it's necessary to run `yarn backend` before testing these changes) The unit tests also ensure that the command removes the proper directories, so there isn't really a need to manually test it, although the reviewer may choose to do so to be safe. Although out of scope for unit tests on this function, we can also do integration tests, to make sure that running the clear command doesn't leave the sourcecred directory in an invalid state from the perspective of the `load` command. ```js $ yarn backend; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --cache; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --all; $ node bin/sourcecred.js load sourcecred/example-github; ``` The expected behavior of the above command block is that the load command never fails or throws an error. @decentralion and I discussed the scenario where `rimraf` errors. We decided that testing this scenario wasn't necessary, because `rimraf` doesn't error if a directory doesn't exist, and rimraf's maintainer suggests [monkey-patching the fs module] to get rimraf to error in testing scenarios. Thanks @decentralion for reviewing and pair-programming this with me. [monkey-patching the fs module]: https://github.com/isaacs/rimraf/issues/31#issuecomment-29534796
2019-05-13 09:59:58 +00:00
- Add the `clear` command (#1111)
- Add description tooltips for node and edge types in the weight configuration UI (#1081)
- Add the `export-graph` command (#1110)
- Enable loading private repositories (#1085)
- Enable setting type weights to 0 in the UI (#1005)
- Add support for 🚀 and 👀 reaction types (#1068)
- Create one page per project, rather than having a selector (#988)
<!-- Please add new entries to the _top_ of this section. -->
## [0.2.0]
Add CLI command to clear sourcecred data directory (#1111) Resolves #1067 Adds the CLI commands: `sourcecred clear --all` -- removes the $SOURCECRED_DIRECTORY `sourcecred clear --cache` -- removes the cache directory `sourcecred clear --help` -- provides usage info `sourcecred clear` -- prompts the user to be more specific Test plan: The unit tests ensure that the command is properly wired into the sourcecred CLI, including help text integration. However, just to be safe, we can start by verifying that calling `sourcecred` without arguments lists the `clear` command as a valid option, and that calling `sourcecred help clear` prints help information. (Note: it's necessary to run `yarn backend` before testing these changes) The unit tests also ensure that the command removes the proper directories, so there isn't really a need to manually test it, although the reviewer may choose to do so to be safe. Although out of scope for unit tests on this function, we can also do integration tests, to make sure that running the clear command doesn't leave the sourcecred directory in an invalid state from the perspective of the `load` command. ```js $ yarn backend; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --cache; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --all; $ node bin/sourcecred.js load sourcecred/example-github; ``` The expected behavior of the above command block is that the load command never fails or throws an error. @decentralion and I discussed the scenario where `rimraf` errors. We decided that testing this scenario wasn't necessary, because `rimraf` doesn't error if a directory doesn't exist, and rimraf's maintainer suggests [monkey-patching the fs module] to get rimraf to error in testing scenarios. Thanks @decentralion for reviewing and pair-programming this with me. [monkey-patching the fs module]: https://github.com/isaacs/rimraf/issues/31#issuecomment-29534796
2019-05-13 09:59:58 +00:00
- Cache GitHub data, allowing for incremental and resumable loading (#622)
- Hyperlink Git commits to GitHub (#887)
- Relicense from MIT to MIT + Apache-2 (#812)
- Display short hash + summary for commits (#879)
- Hyperlink to GitHub entities (#860)
- Add GitHub reactions to the graph (#846)
- Detect references to commits (#833)
- Detect references in commit messages (#829)
- Add commit authorship to the graph (#826)
- Add `MentionsAuthor` edges to the graph (#808)
## [0.1.0]
Add CLI command to clear sourcecred data directory (#1111) Resolves #1067 Adds the CLI commands: `sourcecred clear --all` -- removes the $SOURCECRED_DIRECTORY `sourcecred clear --cache` -- removes the cache directory `sourcecred clear --help` -- provides usage info `sourcecred clear` -- prompts the user to be more specific Test plan: The unit tests ensure that the command is properly wired into the sourcecred CLI, including help text integration. However, just to be safe, we can start by verifying that calling `sourcecred` without arguments lists the `clear` command as a valid option, and that calling `sourcecred help clear` prints help information. (Note: it's necessary to run `yarn backend` before testing these changes) The unit tests also ensure that the command removes the proper directories, so there isn't really a need to manually test it, although the reviewer may choose to do so to be safe. Although out of scope for unit tests on this function, we can also do integration tests, to make sure that running the clear command doesn't leave the sourcecred directory in an invalid state from the perspective of the `load` command. ```js $ yarn backend; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --cache; $ node bin/sourcecred.js load sourcecred/example-github; $ node bin/sourcecred.js clear --all; $ node bin/sourcecred.js load sourcecred/example-github; ``` The expected behavior of the above command block is that the load command never fails or throws an error. @decentralion and I discussed the scenario where `rimraf` errors. We decided that testing this scenario wasn't necessary, because `rimraf` doesn't error if a directory doesn't exist, and rimraf's maintainer suggests [monkey-patching the fs module] to get rimraf to error in testing scenarios. Thanks @decentralion for reviewing and pair-programming this with me. [monkey-patching the fs module]: https://github.com/isaacs/rimraf/issues/31#issuecomment-29534796
2019-05-13 09:59:58 +00:00
- Organize weight config by plugin (#773)
- Configure edge forward/backward weights separately (#749)
- Combine "load graph" and "run pagerank" into one button (#759)
Store GitHub data gzipped at rest (#751) Summary: We store the relational view in `view.json.gz` instead of `view.json`, taking advantage of the isomorphic `pako` library for gzip encoding and decoding. Sample space savings (note that post bodies are included; i.e., #747 has not been applied): SAVE OLD (B) NEW (B) REPO 89.7% 25326 2617 sourcecred/example-github 82.9% 3257576 555948 sourcecred/sourcecred 85.2% 11287621 1665884 ipfs/js-ipfs 88.0% 20953425 2520358 gitcoinco/web 84.4% 38196825 5951459 ipfs/go-ipfs 84.9% 205770642 31101452 tensorflow/tensorflow <details> <summary>Script to generate space savings output</summary> ```shell savings() { printf '% 7s % 11s % 11s %s\n' 'SAVE' 'OLD (B)' 'NEW (B)' 'REPO' for repo; do file="${SOURCECRED_DIRECTORY}/data/${repo}/github/view.json.gz" if ! [ -f "${file}" ]; then printf >&2 'warn: no such file %s\n' "${file}" continue fi script="$(sed -e 's/^ *//' <<EOF repo = '${repo}' pre_size = $(<"${file}" gzip -dc | wc -c) post_size = $(<"${file}" wc -c) percentage = '%0.1f%%' % (100 * (1 - post_size / pre_size)) p = '% 7s % 11d % 11d %s' % (percentage, pre_size, post_size, repo) print(p) EOF )" python3 -c "${script}" done } ``` </details> Closes #750. Test Plan: Comparing the raw old version with the decompressed new version shows that they are identical: ``` $ <~/tmp/sourcecred/data/sourcecred/example-github/github/view.json \ > shasum -a 256 - 63853b9d3f918274aafacf5198787e18185a61b9c95faf640a1e61f5d11fa19f - $ <~/tmp/sourcecred/data/sourcecred/example-github/github/view.json.gz \ > gzip -dc | shasum -a 256 63853b9d3f918274aafacf5198787e18185a61b9c95faf640a1e61f5d11fa19f - ``` Additionally, `yarn test --full` passes, and `yarn start` still loads data and runs PageRank properly. wchargin-branch: gzip-relational-view
2018-09-01 17:42:30 +00:00
- Store GitHub data compressed at rest, reducing space usage by 68× (#750)
- Improve weight sliders display (#736)
- Separate bots from users in the UI (#720)
- Add a feedback link to the prototype (#715)
- Support combining multiple repositories into a single graph (#711)
- Normalize scores so that 1000 cred is split amongst users (#709)
- Stop persisting weights in local store (#706)
Retry GitHub queries with exponential backoff (#699) Summary: This patch adds independent exponential backoff to each individual GitHub GraphQL query. We remove the fixed `GITHUB_DELAY_MS` delay before each query in favor of this solution, which requires no additional configuration (thus resolving a TODO in the process). We use the NPM module `retry` with its default settings: namely, a maximum of 10 retries with factor-2 backoff starting at 1000ms. Empirically, it seems very unlikely that we should require much more than 2 retries for a query. (See Test Plan for more details.) This is both a short-term unblocker and a good kind of thing to have in the long term. Test Plan: Note that `yarn test --full` passes, including `fetchGithubRepoTest.sh`. Consider manual testing as follows. Add `console.info` statements in `retryGithubFetch`, then load a large repository like TensorFlow, and observe the output: ```shell $ node bin/sourcecred.js load --plugin github tensorflow/tensorflow 2>&1 | ts -s '%.s' 0.252566 Fetching repo... 0.258422 Trying... 5.203014 Trying... [snip] 1244.521197 Trying... 1254.848044 Will retry (n=1)... 1260.893334 Trying... 1271.547368 Trying... 1282.094735 Will retry (n=1)... 1283.349192 Will retry (n=2)... 1289.188728 Trying... [snip] 1741.026869 Ensuring no more pages... 1742.139978 Creating view... 1752.023697 Stringifying... 1754.697116 Writing... 1754.697772 Done. ``` This took just under half an hour, with 264 queries total, of which: - 225 queries required 0 retries; - 38 queries required exactly 1 retry; - 1 query required exactly 2 retries; and - 0 queries required 3 or more retries. wchargin-branch: github-backoff
2018-08-22 18:37:29 +00:00
- Execute GraphQL queries with exponential backoff (#699)
Re-introduce a simplified git plugin (#685) This commit re-introduces the git plugin, now that it has been radically simplified as described in [1]. The new git plugin only has nodes for commits and only has commit has-parent edges. As compared to the version that was removed in #628, this plugin is far leaner. It doesn't bloat the graph (for `sourcecred/sourcecred`, the git plugin data is just 164k), and as such doesn't incur much performance penalty. Re-incorporating the git plugin also brings some tangible benefits. We already had git nodes in the graph, as the GitHub plugin attaches them to pull requests. Without any git plugin, these nodes are displayed as "uknown nodes" with ugly descriptions. Also, including a git plugin, even one that is very minimal, communicates to users that git is a source of information to SourceCred, and that they can expect more from it in the future. Note that this commit breaks backcompat for existing repositories that were locally loaded after #628. As such, it is best to `rm -rf $SOURCECRED_DIRECTORY` and start with fresh data. Also, due to a known bug in the WeightConfig, you should reset your browser's local storage. Test plan: After removing the SourceCred directory and the stale localStorage, the cred explorer nicely displays git commits, and connects them via has_parent edges. The NodeType filter allows filtering to commits as expected, and the WeightConfig shows node and edge weights for the Git plugin's nodes and edges. [1]: https://github.com/sourcecred/sourcecred/issues/627#issuecomment-413435447
2018-08-16 20:20:41 +00:00
- Introduce a simplified Git plugin that only tracks commits (#685)
- Rename cred explorer table columns (#680)
- Display version string in the app's footer
- Support hosting SourceCred instances at arbitrary gateways, not just
the root of a domain (#643)
- Aggregate over connection types in the cred explorer (#502)
- Start tracking changes in `CHANGELOG.md`