PR #1325 introduced a failing snapshot test, which was promptly caught
by @wchargin. This commit fixes it by running
`./scripts/update_snapshots.sh`. Also, I bumped the project JSON version
number, which also should have happened in #1325.
Test plan: `yarn test --full` passes.
This commit modifies `cli/load` to appropriately load a Discourse key
from the environment, if it is available.
The mechanics are basically the same as with the GitHub token.
Test plan: Unit tests added. `yarn test` passes.
This commit modifies the `Project` type so that it allows settings for a
Discourse server, and ensures that `api/load` will appropriately load
the server in question, and include it in the output graph.
Putting the full Discourse declaration directly into the Project type is
an unsustainable development practice—in general, adding plugins should
not require changing core data types. However, at the moment I'm punting
on polishing the plugin system, in favor of adding the Discourse plugin
quickly, so I just put it into Project alongside the repo ids.
In the future, I expect to refactor the plugins around a much cleaner
interface; it's just not a priority as yet. (Tracking: #1120.)
This commit also makes the GitHub token optional in `api/load`, since
now it's very plausible that a user will want to only load a Discourse
server, and therefore not require a GitHub token.
As of this commit, it's still impossible to load Discourse projects, as
the CLI always sets a null Discourse server; and in any case, the
frontend would not properly display the project in question, as any
Discourse types would get filtered out.
Test plan: Mocking unit tests have been added to `api/load.test.js` to
ensure that the Discourse graph is loaded and merged correctly.
This adds a new method called `filter` to the `NullUtil` module.
`filter` enables you to filter all the null-like values out of an array
in a convenient typesafe way. (It's really just a wrapper around
`Array.filter((x) => x != null)` with a type signature.)
Test plan: Unit tests added (for both functionality and type safety).
This is the analogue to `github/loadGraph`, but for Discourse. It
basically pipes together the mechanisms for loading Discourse data and
creating a Discourse graph from them, resulting in a single endpoint for
consumption in the API.
In contrast to github, the method is called `loadDiscourse` and not
`loadGraph`, which seemed more appropriate to me. I haven't changed
the corresponding GitHub method's name. (I'm currently knowingly letting
conceputal debt accumulate around the plugin interface; I expect to do a
full refactor within the next few months.)
Test plan: This is the kind of "pipe together tested APIs involving IO"
code which I have decided not to write explicit tests for. However, it
is still protected by flow, and I have a branch (`discourse-plugin`)
which uses this code to do a full Discourse load.
This implements rate limiting to the Discourse fetch logic, so that we
can actually load nontrivial servers without getting a 529 failure.
We could have used retry; I thought it was more polite to actually limit
the rate at which we make requests. However, to avoid seeing 529s in
practice, I left a bit of a buffer: we make only 55 requests per minute,
although 60 would be allowed.
If we want to improve Discourse loading time, we could boost up to the
full 60 request/min, but add in retries. (Or we could switch to retries
entirely.)
Test plan: This logic is untested, however my full discourse-plugin
branch uses it to do full Discourse loads without issue.
Summary:
In #1194, we upgraded Prettier from 1.13.4 to 1.18.2, but this upgrades
past <https://github.com/prettier/prettier/pull/5647>, which was first
released in Prettier 1.16.0. This commit fixes the uses of deprecated
code introduced as a result. It also upgrades the type definitions to
match, via `flow-typed install prettier@1.18.2`.
Addresses part of #1308.
Test Plan:
Prior to this commit, running `yarn unit` would print
```
console.warn node_modules/prettier/index.js:7934
{ parser: "babylon" } is deprecated; we now treat it as { parser: "babel" }.
```
in two test cases; it no longer prints any such warnings. Furthermore,
running `git grep 'parser.*babylon'` no longer finds any matches.
wchargin-branch: prettier-deprecations
Summary:
A `PluginDeclaration` must have a `nodePrefix` and an `edgePrefix`, but
the Discourse plugin declaration was missing these. This was not caught
by Flow because `deep-freeze` was introduced in #1249 without type
definitions; see #1308.
Test Plan:
Apply the following patch:
```diff
diff --git a/src/plugins/discourse/declaration.js b/src/plugins/discourse/declaration.js
index 246a0a28..36ae5f13 100644
--- a/src/plugins/discourse/declaration.js
+++ b/src/plugins/discourse/declaration.js
@@ -1,6 +1,6 @@
// @flow
-import deepFreeze from "deep-freeze";
+declare function deepFreeze<T>(x: T): T;
import type {PluginDeclaration} from "../../analysis/pluginDeclaration";
import type {NodeType, EdgeType} from "../../analysis/types";
import {NodeAddress, EdgeAddress} from "../../core/graph";
```
Note that, with this patch, `yarn flow` fails before this change but
passes after it. Running `yarn unit` still passes.
wchargin-branch: discourse-plugin-prefixes
Summary:
All links in SourceCred must use the `Link` component, providing either
an external URL `href={…}` or an internal route `to={…}`. Any uses of a
raw `<a>` element for internal routes will incur 404s when the
application is hosted on a non-root path, as is currently the case on
the production website.
The change to `FileUploader` is not strictly necessary, as the link has
no styled text and uses a `data:` URL, but there’s no reason not to.
Fixes#1304.
Test Plan:
Build the static site:
```
scripts/build_static_site.sh --target cred --project sourcecred/example-github
```
Then run `python3 -m http.server` from the repository root directory—not
the `cred/` subdirectory—and navigate to the timeline cred view:
<http://localhost:8000/cred/timeline/sourcecred/example-github/>
Observe that the “(legacy)” link now has the correct styling and
correctly navigates to the legacy mode page when clicked: prior to this
change, it would navigate to a URL without the proper `/cred/` path
prefix, yielding a 404. On the legacy page, verify that the “timeline
mode” link has the same properties.
Then, visit <http://localhost:8000/cred/test/FileUploader/> and verify
that the inspection test still passes.
Added a regression test to catch further such errors. Note that
reverting the code changes in this commit causes the test to fail, and
that running it with `--verbose` prints the problematic files.
wchargin-branch: fix-bad-routing-404s
Summary:
This is firing on a production page load of the “prototype” link from
the homepage, and does not seem to actually be an error condition.
Test Plan:
Run `yarn start`, navigate to `/timeline/sourcecred/example-github/`,
and observe that the console error has disappeared.
wchargin-branch: defaultloader-console-error
Summary:
When inserting a “like” action with `INSERT OR IGNORE` semantics, we
also learn whether the action had any effect. We can use this bit to
avoid a separate query checking whether the “like” already exists.
As mentioned here:
<https://github.com/sourcecred/sourcecred/pull/1298#discussion_r314994911>
Test Plan:
Running `yarn test` passes as is, and fails if you change `addLike` to
always return either `changed: true` or `changed: false`.
wchargin-branch: discourse-likes-one-query
Summary:
Calling `db.prepare(sql)` parses the text in `sql` and compiles it to a
prepared statement. This takes time, both for the parsing and allocation
itself and for the context switch from JavaScript to C (SQLite).
Prepared statements are designed to be invoked multiple times with
different bound values. This commit factors prepared statement creation
out of loops so that each call to `update` prepares only a constant
number of statements.
In doing so, we naturally factor out some light JS abstractions over the
raw SQL: `addTopic((topic: Topic))`, rather than `addTopicStmt.run(…)`.
In principle, these could be factored out of `update` entirely to
properties set on the class at initialization time, but, as described in
a comment on the GraphQL mirror, we defer this optimization for now as
it introduces additional complexity.
Test Plan:
Running `yarn test --full` passes.
wchargin-branch: discourse-sql-cse
This commit changes the Discourse default weights around, mostly
significantly moving many weights (e.g. LIKES) that have a 0 backward
weight to have a small positive backward weight instead, like 1/16. In
practice, this mitigates an issue where users with few outbound edges
act as "cred sinks" because the cred gets stuck in a loop between the
user and content they've authored.
Test plan: In local experimentation, I've found the new weights produce
more reasonable-seeming cred attribution.
I've written the Discourse plugin with distinct edge types for post and
topic authorship; it allows us to have more precise control over how
cred flows (and mitigates the need for #968). However, I gave the two
types the same name, which is confusing in the weight config ui. Now
they are properly distinct.
Test plan: It's a simple string change. In (unpublished) commits with a
full Discourse integration, the new strings show up nicely in the UI.
The previous code incorrectly constructed a Discourse post url based on
the post's id, rather than its index within the containing topic. This
is now fixed.
Test plan: There isn't actually a snapshot diff, because the post with
id 2 is also the second post in its thread. I'm not too worried about
this, though: this kind of code changes infrequently, and it's pretty
obvious when it's wrong.
The Discourse mirror class now keeps an up-to-date record of all of the
likes within an instance. It does this by iterating over every user in
the history, and requesting their likes. If at any point we hit a like
we've already seen, we move on to the next user. In the future, we can
improve this so we only query users we haven't checked in a while, or
users who were recently active.
Test plan: Tests verify that we correctly store all the likes, including
after partial updates, and that we don't issue unnecessary queries.
This is a minor change to the Discourse mirror so that it supports a
query to get all users from the server. It will be convenient for a
followon change which makes `update` search for every user's likes.
I also modified createGraph so that it uses the new method, which
results in code that is cleaner and slightly more efficient.
Test plan: Unit tests updated.
For the Discourse plugin, we really want to be able to add a full record
of all of the users' liked posts as edges in the graph. It's a really
high-signal way to move cred, that also gives individual users a lot of
agency and way to engage.
However: we need an API to get this data. Initial searches of API docs
were un-promising; fisrt, we would need to query potentially every post
to get its likes individually (makes it very expensive to find the likes
on old posts), and second, the likes did not come with timestamp
information. For a while, I thought we were at an impasse.
I then went fishing in the Discourse implementation for a solution (yay
open source!). Lots of the API is un-documented, since it's whatever
they happen to add to run Discourse. And it turns out there's a
`user_actions` API ([source]) which can provide all of a user's actions
in order, and having your content liked by someone else is considered an
action. Best of all, these actions come with timestamps.
The upshot is that instead of querying every post to get its likes, we
can query every user to get likes. Iterating over all users can still
be slow, but it's far better than iterating over all posts; plus we can
implement caching so that we only infrequently check in on inactive
users.
I've added a `likesByUser` method to the Discourse fetch interface that
provides this information. I've also added a snapshot test for it (and
updated all of the snapshots). I also rolled in a slight refactor to
error handling in the fetcher.
The mirror doesn't yet use this information (will come later).
[source]: 82e07cb0f4/app/controllers/user_actions_controller.rb (L3)
Test plan: `yarn test` passes. Snapshots look good.
This commit adds the logic needed for creating a contribution graph
based on the Discourse data. We first have a declaration with
specifications for the node and edge types in the plugin. We also have a
`createGraph` module which creates a conformant graph from the Mirror
data. The graph creation is thoroughly tested.
Test plan: Inspect unit tests, run `yarn test`. I also have (yet
unpublished) code which loads the graph into the UI, and it appears
fine.
This is a quick fixup so that the coming createGraph module can be
properly tested.
Shout out to @Beanow for anticipating this need in a [review comment].
[review comment]: https://github.com/sourcecred/sourcecred/pull/1266#discussion_r314305108
Test plan: trivial refactor, run `yarn test`
The mirror wraps a SQLite database which will store all of the data we
download from Discourse.
On a call to `update`, it downloads new data from the server and stores
it. Then, when it is asked for information like the topics and posts, it
can just pull from its local copy. This means that we don't need to
re-download the content every time we load a Discourse instance, which
makes the load more performant, more robust to network failures, etc.
Thanks to @wchargin, whose work on the GraphQL mirror for GitHub (#622)
inspired this mirror.
Test plan: I've written unit tests that use a mock fetcher to validate
the update logic. I've also used this to do a full load of the real
SourceCred Discourse instance, and to create a corresponding graph
(using subsequent commits).
Progress towards #865.
The `DiscourseFetcher` class abstracts over fetching from the Discourse
API, and post-processing and filtering the result into a form that's
convenient for us.
Testing is a bit tricky because the Discourse API keys are sensitive
(they are admin keys) and so I'm reluctant to commit them, even for our
test instance. As a workaround, I've added a shell script which
downloads some data from the SourceCred test instance, and saves it with
a filename which is an encoding of the actual endpoint. Then, in
testing, we can use a mocked fetch which actually hits the snapshots
directory, and thus validate the processing logic on "real" data from
the server. We also test that the fetch headers are set correctly, and
that we handle non-200 error codes appropriately.
Test plan: In addition to the included tests, I have an end-to-end test
which actually uses this fetcher to fully populate the mirror and then
generate a valid SourceCred graph.
This builds on API investigations
[here](https://github.com/sourcecred/sourcecred/issues/865#issuecomment-478026449),
and is general progress towards #865. Thanks to @erlend-sh, without whom
we wouldn't have a test instance.
Summary:
In ES6, the [`try` statement grammar][1] requires a catch parameter; the
parameter is only optional in the latest draft of ECMAScript, which is
of course not yet ratified as any actual standard.
Even though we don’t officially pledge to support Node 8, this is
currently the only breakage, and it’s easy enough to fix.
[1]: https://www.ecma-international.org/ecma-262/6.0/#sec-try-statement
Test Plan:
Running `yarn start` on Node v8.11.4 no longer raises a syntax error.
wchargin-branch: catch-parameter
Summary:
Introduced in #1277.
Test Plan:
Run `yarn start` and visit <http://localhost:8080/test/FileUploader/>.
Conduct the test plan as specified on that page.
wchargin-branch: fileuploader-target
Summary:
Backticks are discouraged relative to the `$(…)` form for command
substitution, because they are harder to read and do not nest without
exponential escaping:
```shell
$ foo=$(echo $(echo hi $(echo bye))) # clear
$ foo=`echo \`echo hi \\\`echo bye\\\`\`` # hmm
```
In this context, command substitution should not be used at all; `PWD`
is a special variable that always contains the current working
directory. The new version of the code will be correct even if the
current working directory ends with whitespace that would be stripped
off by the command substitution.
Test Plan:
Prepended an `echo` to the relevant line, and verified that the script
has the same output before and after this change.
wchargin-branch: fix-backticks
The code is mostly ported from the legacy app. However, we no longer
assume that we are showing every type for every plugin. Instead, the
types are manually selected. For now, we permit the GitHub user type,
and the GitHub repo type, as these are the two types that are included
in filtered timeline cred.
Test plan: Manual inspection is necessary, since this frontend is mostly
untested. I've done that inspection. Also, `yarn test` passes.
Minor change to the API for MapUtil.pushValue. Now it returns the
resultant array. I've found this convenient in at least one case, and
previously we weren't returning anything, so it's a cheap change.
Test plan: Unit test added.
I moved sourcecred/example-git{,hub} to the @sourcecred-test org.
This commit fixes the build given that move.
I've realized that in #1233 I in-advertently made some Git tests that
depend on a snapshot un-updateable. I'm going to compound on that slight
technical debt by skipping the tests that depended on that snapshot. I
recognize and accept that I'll need to pay this down when I resuscitate
the git plugin.
Test plan: `yarn test --full`.
This commit removes the `pagerank` and `analyze` commands (both of which
never saw real usage), removes the outdated adapter-based `loadGraph`
method, and removes all traces of the analysis adapters.
It builds on work in #1233 and #1136.
Test plan: `yarn test --full` passes.
This commit swaps usage over to the new implementation of `cli/load`
(the one that wraps `api/load`) and makes changes throughout the project
to accomodate that we now track instances by Project rather than by
RepoId.
Test plan: Unit tests updated; run `yarn test --full`. Also, for safety:
actually load a project (by whole org, why not) and verify that the
frontend still works.
I'm re-organizing SC data to be oriented on the graph, rather than on
plugin-specific data structures. So there is no longer a need for the
git loading logic which orients around saving a repository.json file
that's been potentially merged across repos, or indeed the logic for
merging repositories at all. So I'm removing `git/loadGitData`,
`git/mergeRepository`, and relatives.
Test plan: `yarn test --full` passes.
There's no need for us to depend on `mkdirp`, because the `fs-extra`
module already has `fs.mkdirp` and `fs.mkdirpSync`. This commit removes
the dep from our `package.json`, and removes all explicit imports of it.
Test plan: `yarn test --full` passes. `git grep "import mkdirp"` has no
hits.
This adds a new module, `api/load`, which implements the logic that will
underly the new `sourcecred load` command. The `api` package is a new
folder that will contain the logic that powers the CLI (but will be
callable directly as we improve SourceCred). As a heuristic, nontrivial
logic in `cli/` should be factored out to `api/`.
In the future, we will likely want to refactor these APIs to
make them more atomic/composable. `api/load` does "all the things" in
terms of loading data, computing cred, and writing it to disk. I'm going
with the simplest approach here (mirroring existing functionality) so
that we can merge #1233 and realize its many benefits more easily.
This work is factored out of #1233. Thanks to @Beanow for [review]
of the module, which resulted in several changes (e.g. organizing it
under api/, having the TaskReporter be dependency injected).
[review]: https://github.com/sourcecred/sourcecred/pull/1233#pullrequestreview-263633643
Test plan: `api/load` is tested (via mocking unit tests). Run `yarn test`
This commit refactors the `util/taskReporter` module so that
`TaskReporter` is an interface; the class previously called
`TaskReporter` is renamed to `LoggingTaskReporter`. We also export a
`TestTaskReporter` which implements the interface, and is very easy to
test.
The motivation: This will make it much easier to write tested code that
uses a `TaskReporter`, as now the test code can provide a
`TestTaskReporter` and check that all tasks get finished, that task
ids are as expected, etc.
Test plan: The `TestTaskReporter` is tested. Run `yarn test`.
Throughout the codebase, we freeze objects when we want to ensure that
their properties are never altered -- e.g. because they are a plugin
declaration, or are being re-used for various test cases.
We generally use `Object.freeze`. This has the disadvantage that it does
not work recursively, so a frozen object's mutable fields and properties
can still be mutated. (E.g. if `const obj = Object.freeze({foo: []})`,
then `obj.foo.push(1)` will succeed in mutating the 'frozen' object).
Sometimes we anticipate this and explicitly freeze the sub-fields (which
is tedious); sometimes we forget (which invites errors). This change
simply replaces all instances of Object.freeze with [deep-freeze], so we
don't need to worry about the issue at all anymore.
Test plan: `yarn test` passes (after updating snapshots);
`git grep Object.freeze` returns no hits.
[deep-freeze]: https://www.npmjs.com/package/deep-freeze
This is a replacement for `github/loadGithubData` which returns a
combined Graph rather than a combined RelationalView. This provides a
major benefit, which is that we can use the (robust) Graph merge logic
rather than the (buggy) relational view merge.
Test plan: This function is untested. It basically pipelines a few APIs
together. I think that flow is basically sufficient to validate that it
works, and writing a unit test will be frustrating (mostly will involve
re-integrating the funcitonality via mocks). A future commit makes this
part of the pipeline that generates snapshot tests, so it is de-facto
integration tested.
This module builds on the project logic added in #1238, and makes it
easy to create projects based on a simple string configuration.
Basically, the spec `foo/bar` creates a project containing just the repo
foo/bar, and the spec `@foo` creates a project containing all of the
repos from the user/organization named foo.
This is pulled out of #1233, but I've enhanced it to support
organizations out of the box.
The method is thoroughly tested.
Test plan: `yarn test --full` still passes. Also, I've ensured that the
async `_getProjectIds` is still usable in our webpack configs (via
modifying and testing the dependent commits).
This creates a new `Project` type which will replace `RepoId` as the
index type for saving and loading data.
The basic data type is added to `project.js`. Rather than having a
`RepoIdRegistry`, I intend to infer the registry at build time by
scanning for available projects saved in the sourcecred directory. I've
added the `project_io` module for this task. It has methods for setting
up a project subdirectory, and loading the `Project` info from that
subdirectory.
To ensure that projects ids can be encoded even if they have symbols
like `/` and `@`, we base64 encode them.
To ensure that project ids can be retrieved at build time, the
`getProjectIds` method is factored out into its own plain ECMAScript
module. For all non-build time needs, it is re-exported from
`project_io`.
Test plan: Unit tests added; run `yarn test`.
See #1243 for context. This is basically a more aggressive version of
pull #1230 -- instead of just running unit tests in isolation, we also
run flow in isolation, and kill the servers afterwards.
Test plan: See how this fares in CI :)
Includes a change to `cli/load` and `build_static_site.sh` to accept a `--weights WEIGHTS_FILE` argument.
This allows overriding the default weights at build-time using a `weights.json` that has the same format as previously generated in the frontend.
Test plan:
Adds an additional test-case as well for propagating the optional parameter.
The file I/O of loading and parsing a weights.json file was tested manually. As analysis/weights' fromJSON() is tested elsewhere as is passing weight parameters.
Prior to #1136, we needed an `ExplorerAdapter` abstraction to get node
description data to the frontend. Now that it's included in the graph,
we can throw away this abstraction, which is a big step towards plugin
simplification (work towards #1120).
Since it only affects a deprecated/legacy part of the code base, I
didn't put much effort into making the result super clean. I also
removed a few tests that became inconvenient.
Test plan: Verified that the legacy frontend still works. There's one
tiny regression, which is that the link color in the legacy frontend no
longer matches the rest of the UI, but that's actually consistent with
the timeline frontend, so no biggie.
`yarn test` passes.
The scores are lightly processed from their internal representation.
Example usage:
```
$ yarn backend;
$ node bin/sourcecred.js load sourcecred/sourcecred
$ node bin/sourcecred.js scores sourcecred/sourcecred > scores.json
```
The data structure is as follows:
```js
export type NodeOutput = {|
+id: string,
+totalCred: number,
+intervalCred: $ReadOnlyArray<number>,
|};
export type ScoreOutput = Compatible<{|
+users: $ReadOnlyArray<NodeOutput>,
+intervals: $ReadOnlyArray<Interval>,
|}>;
```
Test plan: I added sharness tests at `sharness/test_cli_scores.t`.
In the past, we've used javascript tests for CLI commands. However,
those are pretty time-consuming to write, and are less robust than
simply running the command from bash. Check the snapshot for a sense of
what the new data format looks like. Also, the snapshot updater now
updates this snapshot too.
Relevant for #1047.
Thanks to @Beanow for feedback on the output format and design.
Thanks to @wchargin for help in code review.
As of this commit, the main SourceCred prototypes page now links to
timeline cred, meaning that timeline cred is now live. I've added a
link from the legacy explorer to the timeline explorer (which already
has a link out to the legacy explorer).
Test plan: Careful inspection of the frontend by the committer.
Also, yarn test.
This is a bulk rename of all the old explorer code into
`explorer/legacy`. Now that the timeline explorer exists, I intend to
prioritize development on that going forwards. Once the timeline
explorer is as good as the old explorer at decomposing a node's sources
of cred, I will remove the legacy explorer entirely.
Test plan: `yarn test`
This commit adds a TimelineExplorer for visualizing timeline cred data.
The centerpiece is the TimelineCredChart, a d3-based line chart showing
how the top users' cred evolved over time. It has features like tooltips,
reasonable ticks on the x axis, a legend, and filtering out line
segments that stay on the x axis.
An inspection test is included, which you can check out here:
http://localhost:8080/test/TimelineCredView/
Also, you can run it for any loaded repository at:
http://localhost:8080/timeline/$repoOwner/$repoName
This commit also includes new dependencies:
- recharts (for the charts)
- react-markdown (for rendering the Markdown descriptions)
- remove-markdown (so the legend will be clean text)
- d3-time-format for date axis generation
- d3-scale and d3-scale-chromatic for color scales
Test plan: The frontend code is mostly untested, in keeping with my
observation that the costs of testing the old explorer were really high,
and the tests brought little benefit. However, I have manually tested it
thoroughly. Also, there is an inspection test for the TimelineCredView
(see above).
It's very simple: a method that creates a copy of a `Weights`.
While writing this, I realized I should probably refactor the weights
module so that it exports a class rather than a bunch of methods
operating on a data structure. It would just be a cleaner API. But I'm
leaving that for another day.
Test plan: Unit tests added.
As described in #987, we use a single TTL across GitHub types. Right
now, the TTL is set to 7 days. This means that it's possible to run
`sourcecred load`, but still be missing the last 7 days worth of issues.
Now that we're doing timeline cred (cf #1212), this is not acceptable.
As a workaround until we fix#987, I'm decreasing the TTL to 12 hours.
That's still long enough to make a good experience for someone who is
tweaking config and calling `sourcecred load` a lot, but ensures that
freshly-loaded results still have recent activity.
Test plan: `yarn test`
This adds a TimelineCred class which serves several functions:
- acts as a view on timeline cred data
- (lets you get highest scoring nodes, etc)
- has an interface for computing timeline cred
- lets you serialize cred along with the graph and paramter settings
that generated it in a single object
One upshot of this design is that now if we let the user provide weights
(or other config) on load time in the CLI, those weights will get
carried over to the frontend, since they are included along with the
cred results.
TimelineCred has 'Parameters' and 'Config'. The parameters are
user-specified and may change within a given instance. The config is
essentially codebase-level configuration around what types are used for
scoring, etc; I don't expect users to be changing this. To keep the
analysis module decoupled from the plugins module, I put a default
config in `src/plugins/defaultCredConfig`; I expect all users of
TimelineCred to use this config. (At least for a while!)
Test plan: I've added some tests to `TimelineCred`. Run `yarn test`. I
also have a yet-unmerged branch that builds a functioning cred display
UI using the `TimelineCred` class.
fixup tlc
This adds the `filterTimelineCred` module, which dramatically reduces
the size of timeline cred by throwing away all nodes that are not a user
or repository. It also supports serialization / deserialization.
Test plan: unit tests included
This module takes the timeline distributions created by
`timelinePagerank`, and re-normalizes the scores into cred. For details
on the algorithm, read comments and docstrings in the module.
Test plan: Unit tests added.
As the name would suggest, this module allows computing timeline
PageRank on a graph. See documentation in the module for details on the
algorithm.
Test plan: The module has incomplete testing. The timelinePagerank
function calls out to iterators for getting the time-decayed node
weights and the time-decayed markov chain; these iterators are tested.
However, the wrapper logic that composes these pieces together,
calculates seed vectors, and runs PageRank is not tested. I felt it
would be a pain to test and settled for reviewing the code carefully,
and putting a cautionary note at the top of the function.
This commit adds new weight evaluators for nodes and edges. Unlike the
previous evaluator, edges and nodes are handled as separate concerns,
rather than composing the node weights into the edge weights. I think
this separation is cleaner.
Both evaluators use only the address, not the full (Node or Edge)
object. Although we may want to give the edge evaluator access to the
full Edge later, if we decide we want node-type-differentiated edge
weights (e.g. if a hasParent edge has a different weight depending on
whether it is connected to an Issue or a Repository).
weightsToEdgeEvaluator has been refactored to use the new evaluators,
and has been given a deprecation notice.
Test plan: `yarn test`
This commit adds an `interval` module which defines intervals (time
ranges), and methods for slicing up a graph into its consistuent time
intervals. This is pre-requisite work for #862.
I've added a dep on d3-array.
Test plan: Unit tests added; run `yarn test`
This commit disables the Git plugin by removing it from the default list
of plugins to load, or to display in the frontend.
Rationale: The git plugin doesn't currently add very much to cred
quality. Git commits have edges to their parent, which isn't a very
meaningful relationship for cred purposes. We'll want to re-enable the
Git plugin once we're ready to support e.g. file and directory level
cred tracking.
I've skipped a block of tests around the git analysisAdapter. (I intend
to deprecate the analysisAdapters, so skipping the tests seemed
preferrable to updating them). I also updated our sharness test for
catching test files without a proper describe block, so that it won't
error on skipped blocks.
Test plan: `yarn test --full` passes. Loading a new repository and
inspecting it in the frontend gives consistent results. There are no
references to Git plugin weights in the frontend, now that corresponding
nodes are not available.
The dependabot bot has an inconsistent typename in GitHub's database.
We'll blacklist it so we can continue loading `sourcecred/sourcecred`.
Test plan: `node bin/sourcecred.js load sourcecred/sourcecred` fails
before this commit, and succeeds after.
This commit resolves an inconsistency where we called edge weights
`toWeight` and `froWeight` in the core/attribution module, but
`forwards` and `backwards` in the analysis module.
I changed field names in the PagerankGraph JSON, so I bumped its compat.
Test plan: `yarn test --full` passes.
This commit just adds a test which verifies that when an
OrderedSparseMarkovChain is created by graphToMarkovChain, its nodeOrder
is the graph's canonical node order.
Test plan: `yarn test`
This means that we no longer need to expose methods for extracting the
order from serialized JSON. We can always count on iterating over the
nodes and edges in sorted order.
Test plan: `yarn test`; tests updated.
This commit updates eslint from v4 to v6. In doing so, I've moved off of
the create-react-app base eslint config. We were on an old version (v2)
and it doesn't make sense to update to v4, as in v4 create-react-app
uses typescript. Also, it didn't make sense to stay on
create-react-app's v2 config, because then it had unmet peer dependency
constraints on old versions of eslint.
Instead, I've moved us to use the default rules for eslint,
eslint-plugin-react, and eslint-plugin-flowtype.
I also made some changes to the codebase to satisfy the new lint rules
that came with this change.
Test plan: `yarn test` passes.
This necessitated a number of type fixes:
- Upgraded the express flow-typed file to latest
- Added manual flow error suppression to where the express flow-typed
file is still using a deprecated utility type
- Removed type polymorphism support on map.merge (see context here[1]).
We weren't using the polymorphism anywhere so I figured it was simplest
to just remove it.
- Improve typing around jest mocks throughout the codebase.
Test plan: `yarn test --full` passes.
[1]: https://github.com/flow-typed/flow-typed/issues/2991
This commit updates our prettier version from `1.13` to `1.18`. Looks
like software does get better over time! I like all of the changes.
Test plan: `yarn test` passes. I've manually inspected the diffs.
When we took a dep on better-sqlite3 in #836, we used a fork, because
better-sqlite3 did not yet support private in-memory databases via the
`:memory:` filepath. As of better-sqlite3 v5, this has been added to
mainline, so we no longer need the fork.
The v4->v5 transition involves some breaking changes. The only ones that
affected us were two field renames, from `lastUpdateROWID` to
`lastUpdateRowid`, and `returnsData` to `reader`.
Test plan:
After updating the field accesses, `yarn test --full` passes. For added
safety, I also blew away cache, loaded a nontrivial repository, and
verified that the full cred workflow still works.
cc @wchargin
This updates the graph `Node` type to include a string description.
The description should be a brief (ideally oneline) string giving
context on what the node is. All planned frontends will support
markdown, so linking to context (e.g. linking to the issue corresponding
to an ISSUE type node) is supported.
This commit updates the Git and GitHub plugins to use the new
description field.
Test plan: `yarn test --full` passes, and I've inspected snapshots and
made sure they look reasonable.
The GitHub plugin no longer adds a Node to the graph for Git commits.
Instead, it creates a dangling edge to it. This frees the GitHub plugin
from responsibility for setting the timestamp or other metadata for Git
nodes.
The Git plugin no longer adds a Commit Node to the graph immediately when
encountering a commit's parent hash. Instead, it creates an edge to the
parent, and then fills in the parent node once it is encountered in the
commit store.
Test plan: Load a real repository with merged pull requests
(e.g. sourcecred/research) into the explorer, and verify that GitHub
commit entities are still connected to Git commits, and that Git
commits are still connected to their parents.
This commit modifies the Graph class so that it permits dangling edges;
that is to say, edges whose src or dst are not present in the graph.
Dangling edges may be directly added to the graph, or existing edges may
become dangling if their src or dst is removed.
This change is prerequisite to #1136; if we require that nodes have
metadata, we should also make it possible to add edges to nodes that
don't yet exist, as the plugin creating an edge may not have access to
the full metadata needed to add the node.
To support this change, there is now an `isDanglingEdge` method on the
graph, which reports whether or not the edge is dangling. Also,
`Graph.edges` requires that the client make an explicit choice on
whether dangling edges are desired. This ensures that we do not
accidentally include dangling edges in a case where they are
inappropriate (e.g. creating a Markov chain) or accidentally discard
dangling edges when they are needed (e.g. when merging or serializing).
The Graph's invariant checker has been updated to reflect the new
semantics.
The Graph compat version has been bumped, since this is a break in
backwards compatibility.
Note that this commit does not change the behavior of any plugins; that
is to say, no plugins create dangling edges (yet).
Test plan: The advanced graph test case has been updated to include
dangling edges. The tests for Graph, PagerankGraph, and
GraphToMarkovChain have been updated. `yarn test --full` passes.
This commit modifies the base `Graph` class so that nodes are now
represented by `Node` objects rather than `NodeAddressT`. The intention
is to start adding additional fields (e.g. description and timestamp) to
nodes, although that is not included in this commit.
See #1136 for rationale.
Test plan: The graph is very well tested, and this commit adds
additional tests and invariant checking. Some additional test code
needed update. `yarn test --full` passes, and the SourceCred UI works as
expected.
PagerankGraph's `node` and `edge` getters returned null for unavailable
entries, rather than undefined. This is inconsistent with general JS
behavior, and with the base Graph. I've now cleaned it up.
Test plan: unit tests updated; `yarn test` passes.
Every GitHub entity now has a `description` method which returns a short
markdown description. These will be added to the graph as part of #1136.
Test plan: Inspected snapshots, `yarn test --full`.
This will allow timeline cred (#862) to do a better job of flowing cred
across reaction edges. (Very old reactions should not be moving a lot of
present-day cred.)
Test plan: Inspected snapshot changes.
Every GitHub entity from `RelationalView` now has a `timestampMs`
method. This replaces the standalone `createdAt` method.
Test plan: Snapshots look good.
It's an extension of #1152 induced by #1175.
It's a very simple change; I just changed the schema, and
`scripts/update_snapshots.sh` took care of the rest.
Test plan: Inspected snapshots and generated flow types.
This pulls distribution related code out of `markovChain.js` into the new
`distribution.js` module, and from `graphToMarkovChain.js` into
`nodeDistribution.js`.
Since the `computeDelta` method is now exported, I've added some unit
tests.
Test plan: `yarn test` passes.
As #1136 will be moving timestamps into the graph, we no longer need
`createdAt` method in the `AnalysisAdapter`. Actually, we no longer need
the adapter/loader distinction introduced in #1157. I haven't taken the
time to remove the `BackendAdapterLoader` concept because a) we may need
it later, and b) if we don't, I'll likely remove the `AnalysisAdapter`
concept entirely, in favor of having plugins directly save `graph.json`
files to a known location.
Test plan: `yarn test` passes.
I added `mentionsAuthorReference` based on an untested hypothesis that
they would be useful. With the passage of time, I've never seen any
evidence that they actually improve cred socres (their impact seems
negligible), and they add complexity.
In the future, "go-fishing" style heuristics like this should not merge
unless they are of clearly demonstrated value. Also, it would be better
to add stuff like this via a standalone plugin rather than in the core
GitHub logic.
Undoes #806.
Test plan: `yarn test`
Now that the graph is saved by default as a part of load, users who need
the graph can grab it directly from the `$SOURCECRED_DIRECTORY`. If we
really need a command line util for grabbing it, we should rewrite that
command to just grab the graph from that spot rather than re-computing
it.
Test plan: `yarn test`
As of the timeline cred work, I'm shifting emphasis away from raw
PageRank results, in favor of timeline pagerank results. As such,
there's no need to have load save the regular pagerank results on
creation.
As of #1136, there will be no need for timestampMap, as that data will
be present directly in the graph. As the timeline cred UI will depend on
the full graph for analysis, let's save the graph instead.
Test plan: `yarn test` and snapshot inspection.
This commit adds new helper methods for creating test nodes (`node` and
`partsNode`) and for creating test edges (`edge` and `partsEdge`) to
graphTestUtil.js.
This is very helpful in light of work related to #1136. I'm going to
change the concept of "node" from a raw address to an object, add fields
to that object, and add fields to the `Edge` type. If done naively, we
would need to change all the test code across the project for every one
of those changes.
By centralizing the creation of test nodes and edges behind the new
functions, we can update all the test code in a single place.
This change is trivial from a conceputal perspective, and very
broad-reaching from a code-touching perspective. It should be easy to
review, because if tests pass then the change is probably working as
intended. :)
Test plan: `yarn test`
In #1132 and #1134, I started work on the Odyssey plugin. However,
before getting it to a state where it's usefully included in SourceCred,
I decided to pivot to focus on timeline cred first.
Now I'm merging significant refactors as a part of timeline cred
(#1136). As a side effect of this refactor, the Odyssey plugin should
undergo significant changes (OdysseyInstance is now basically redundant
with base Graph.) Rather than incrementally update unused code, I elect
to remove the plugin. This code should be revived on a side branch, and
then merged into master once we have a fully functioning prototype.
Test plan: `yarn test` passes.
This commit refactors the Graph class so that rather than having
separate maps for inEdges and outEdges, there is a single incidentEdges
map, which contains objects with inEdges and outEdges.
This is motivated by a forthcoming big change as part of #1136; namely, to
allow storing dangling edges in the graph. Once we do so, we'll need a
consistent source of truth that enumerates all of the node addresses
which are accessible in the graph (either because they correspond to a
node in the graph, or because they are the src or dst of a dangling
edge). We could do this by adding another field to graph which tracks
this set, but by making this refactor, we can instead use the key set of
_incidentEdges as the source of truth for which node addresses are
present.
Besides being motivated by #1136, I think it's cleaner in general. Note
there are fewer ways for the graph to be inconsistent, as it's no longer
possible for inEdges and outEdges to have inconsistent sets of node
addresses.
The most complicated piece of this change was updating the automatic
invariant checker. It was no longer possible to test 3.1 and 4.1
separately, so they needed to be merged into a new invariant. Rather
than re-enumerate the invariants, I called the new one the 'Temporary
Invariant', because it is going to disappear in a subsequent commit.
Test plan: `yarn test` passes. Since Graph has extremely thorough
testing, this gives me great confidence in this commit. Note that no
observable behavior has changed.
At present, the Git commit node type lives in a strange state of shared
responsibility between GitHub and Git. The Git plugin is nominally
responsible for it, but its render method tries to show a hyperlink to
GitHub -- which is awkward for many reasons, including that the same Git
commit could have multiple hyperlinks on GitHub.
This commit resolves that issue by separating the existing commit type
into two: the Git Commit type, which is owned by the Git plugin and
doesn't have hyperlinks or any fancy GitHub metadata, and the GitHub
Commit, which is owned by the GitHub plugin, corresponds to a unique
database id in GitHub, and has a corresponding GitHub url.
The two commits are connected by a CorrespondsToCommit edge type, which
links from the GitHub commit to the corresponding Git commit.
This is necessary for #1136, as if we want to make descriptions a part
of the graph payload, we need for descriptions to be unique for a given
address--and descriptions are only unique if we identifiy each GitHub
commit pointer as a separate address.
Test plan: The unit testing in this part of the codebase is light, so I
verified that the frontend work as expected for `sourcecred/sourcecred`
and `sourcecred/research`. The new node type and edge type appear
properly in the UI, the GitHub commits are connected to their Git
counterparts, etc.
A long time ago, we made graph views for git and github. These are
interfaces over the graph which allow retrieving nodes' relations, e.g.
finding the parent address of a commit just using the graph.
These are fairly complex, and have seen almost no use at all. The one
thing they are used for is implementing invariant checking. The
invariant checking is nice in principle, but since we only apply it to
the example data, its of very limited value in practice.
Since I'm planning a significant Graph refactor (#1136), I'd rather delete this
code than continue to maintain it, since I think it's complexity/value
ratio is unfavorable.
Test plan: `yarn test`
This is another minor silent test failure: the error message thrown by
loadIndividualPlugin when a GitHub token is not available is not quite
the error that was specified in the test.
There were two issues: that we were testing for the wrong error message,
and that the failure didn't fail the test. I fixed both issues (by
changing the message thrown to match the test, and by having the test
_return_ the expectation that the promise will reject, and by expecting
there to be one assertion.)
Test plan: `yarn unit cli/load` no longer shows any unhandled promise
rejection warnings. If the test is modified so that it checks for the
wrong string, it now properly fails rather than passing with an
unhandled rejection.
Prior to this commit, if you run `yarn unit cli/load`,
you would see a lot of unhandled promise rejection warnings related to
the fact that load calls a `saveTimestamps` function which expects the
SourceCred directory to contain the results of really loading SourceCred
plugins. However, when testing, these functions have been mocked, and so
saveTimestamps rejects. This rejection was not caught, and polluted the
test output without actually failing the tests.
This commit updates the tests so that saveTimestamps can be mocked (via
dependency injection) and we can both verify that it is invoked
correctly, and not pollute the test output with spurious warnings.
Test plan:
- `yarn test --full` passes
- `yarn unit cli/load` produces far fewer
UnhandledPromiseRejectionWarnings (there is still one unrelated one)
- loading sourcecred/research still works (as a canary)
Note: the PR which introduced this issue is #1162.
As discussed in #1163, #1162 caused `sourcecred load` to start failing
for real repos (e.g. sourcecred/research). This commit merges a
somewhat hacky fix.
Test plan: `yarn test` passes, and `sourcecred load sourcecred/research`
works.
Fixes#1156
When users export a GitHub API token that has insufficient privleges
or has been revoked, we have been using a catch all error with retry
to handle it. This change adds a new error type for bad credentials
and does not retry.
Test plan:
There are no unit tests that cover this, however, you can test the
change by supplying a revoked token and attempting to load a GitHub
repo.
This modifies `sourcecred load` so that it saves timestamp information
for all of the loaded plugins in a single aggregated map.
This is quite convenient, as it saves consumers of timestamp information
from needing to worry about the (rather hacky) implementation whereby
the data is fed from each adapter. Instead, consumers can just load the
timestamp map. This will also make it much easier to use timestamp info
in the research codebase.
Test plan: The timestampMap module has testing around generating the map
from the adapter and nodes, writing it, and reading it.
I haven't added any testing to the `load` CLI command. I think it would
be redundant as the updated snapshot test reveals that the map is
getting serialized properly.
Tests pass, and I have inspected the snapshot
This commit modifies `PagerankGraph.runPagerank` so that rather than
always starting from a uniform distribution, it starts with the
PagerankGraph's existing score distribution. The PagerankGraph
initializes with a uniform score over nodes, so it has the exact same
behavior on the first time that runPagerank is called. On subsequent
calls, PageRank will likely converge a lot faster, because it's starting
from converged scores. (It should still be a lot faster in cases where
e.g. the user has tweaked the weights.)
In certain degerate cases, this could change the resultant scores.
Specifically, if there are isolated nodes in the graph and alpha=0, then
the isolated nodes' final scores depends on the initial score. In
general, I think this won't be an issue as we expect alpha > 0 in normal
usage.
Test plan: I added a unit test to verify this property, by checking that
running PageRank with maxIterations==0 on an already-converged graph
results in a still-converged graph. Also, existing tests pass.
I think we can now close#1020.
As SourceCred has evolved, we've grown more and more snapshot tests that
are not included in Jest. The GitHub plugin has two ad-hoc snapshot
tests, the Git plugin has one, and the sharness test suites have one.
This makes it difficult to keep track of where to update snapshots when
core changes are made. To fix this, I've added a script,
`scripts/update_snapshots.sh`, which updates snapshot tests across the
project.
Test plan: I removed existing snapshots across the codebase, ran the
snapshot tester, and they correctly regenerated.
* Refactor Loader from AnalysisAdapter
At present, the only data the AnalysisAdapter provides is the Graph, so
the AnalysisAdapter has a `load` method which directly returns the
graph. I'm planning to add a `createdAt` getter to the adapter as well,
which also will depend on loading the data. To make this change
convenient, I'm starting by refactoring an AdapterLoader out, which
manages loading data from disk, so that once we have an AnalysisAdapter,
it already has all relevant data loaded. Then, it will be much easier to
add a `createdAt` method.
Test plan: Tests updated, flow passes.
* Add `createdAt` to the analysis adapter
A big step forward for time-varying cred. This will make `createdAt`
timestamps available for PageRank analysis.
Test plan: Added new unit tests. Inspect the snapshots. Run `yarn test`.
Modifies the Git plugin so that we now track commit author dates.
Similar to in #1152, they are encoded in MsSinceEpoch.
Test plan: `yarn test --full` passes, except for the pre-existing
failure discussed in #1151.
Thanks to @s-ben for a conversation which motivated these changes.
Updates github schema to include createdAt timestamps, and then updates
the RelationalView to provide those timestamps as MsSinceEpoch.
I added createdAt timestamps to Repos, Issues, Pulls, Reviews, and
Comments, as these correspond to GitHub graph nodes where I think
time-based filtering is relevant. I didn't add them to Users, Reactions,
or Commits. Reactions, because they correspond to edges not nodes. (We
could consider doing the time filtering on edges too, but I'd rather
keep it simple for now.) Commits, because they're owned by a different
plugin. Users, because... in a certain sense the user identity is
timeless, the time factoring is mostly so we can evaluate how users'
cred varies over time.
Anyway, it will be easy to add more fields later if we need them.
Test plan:
- Inspect snapshot changes
- Ran `yarn test --full`
- Its only failure is pre-existing, per #1151
Thanks to @s-ben for some motivation and discussion about this change.
This adds a WeightsFileManager component that allows the user to save or
load weights in the cred explorer. Clicking the download icon downloads
the weights, clicking the upload icon uploads them.
I also did a slight refactor to the FileUploader so that it no longer
always provides the file upload icon, instead the instantiator passes
children which act as the upload clickable. Seemed more consistent.
Test plan: No tests added, but I manually tested that upload and
download both work.
* Add FileUploader with inspection test
TODO: get it working
* Add a FileUploader component, with inspection test
This adds a FileUploader component, which allows the user to upload JSON
files. Rather than using automated testing, it has an inspection test.
The inspection test may be run by navigating to:
http://localhost:8080/test/FileUploader/
This commit also adds some basic utility functions for defining
inspection tests to `routeData.js`. We should improve support for
inspection tests in the future; see [#1148].
[#1148]: https://github.com/sourcecred/sourcecred/issues/1148
Test plan: Ran the included inspection test.
This commit refactors the `analysis/weights` module so that there's
a single top-level type called `Weights` which includes all of the
user-specified weights for running PageRank. This includes both manually
specified type weights, and per-node weights.
In contrast to the old weights module which had weighted types (i.e. the
type bundled with the weight), this only records the weight choice
(keyed by address). Similarly, the map is empty unless the user has
explicitly chosen a weight. This has a few advantages:
- For planned work on serializing weights, it's convenient that
there's a single unified object that bundles all the weights.
- For planned work on serializing weights, it's convenient that we only
record actual choices, not the default values (defaults are provided
within the WeightsToEdgeEvaluator). This way, we don't need to manually
filter out the default weights for serialization. (We want users who
have not made any choice about the type weights to automatically get new
type weightings we publish in sourcecred/sourcecred).
- Overall, the WeightConfig data piping has become a lot simpler.
While making these changes, I threw away the PluginWeightConfig
component, instead inlining it in the top-level WeightConfig. I also
moved responsibility for the WeightConfig out of the PagerankTable, and
into the containing App; the WeightConfig is created and wired by the
App, and then passed as a whole component into the PagerankTable via
props. This means that PagerankTable does not need to get a bunch of
extra props for piping the state changes into WeightConfig.
I also threw away some frontend test code for the WeightConfig
component. Over the course of the past few months, I've observed that
the value of a lot of the frontend-wiring-testing is pretty low (i.e.
they are not catching issues, because Flow catches them) and the cost of
maintaining the testing is high. I'm now inclined to believe that we
shouldn't be testing routine component wiring via explicit tests,
because it's very easy to verify that the behavior is correct by
interacting with the UI, and the tests are expensive to write and
maintain.
For "core logic" (e.g. behaviors of types and data structures,
algorithms, etc) I continue to believe testing is very valuable. One
heuristic I'll start applying is: "can I achieve high confidence that
this code is correct, without tests, using little effort by manually
inspecting the frontend?". If the answer is yes, I'm unlikely to write
tests.
Test plan: `yarn test` passes, and the UI continues to function,
including changing manual and type level weights and observing cred
changes.
Node and Edge types are increasingly important in SourceCred, as we use
them to decide what weights to provide, what description logic to use,
etc. In #631, we hit a bug around not finding a type matching a
particular node, and in #640 I responded by creating a
"FallbackAdapter", which matches any node, and a
"Static/DynamicAdapterSet", which are abstractions for finding adapters
in a way that guarantees the presence of the fallback adapter.
I think this solution is actually pretty brittle. It only works if we
are disciplined in only ever accessing node and edge types using a
context that included the FallbackAdapter, and it requires us to
centralize all of the "type-not-found" logic in one place (the fallback
declaration) irrespective of what the locally appropriate solution is.
Looking back at #631, the core issue was that we had a getter that
promised to always give a matching type and adapter, rather than
returning a possibly undefined adapter. We fixed this by attempting to
ensure that there always would be a matching adapter. I think it would
be better to have the methods honestly report that the adapter might be
null/undefined, and let the client code handle it appropriately.
This commit implements that change. It's motivated by me being
frustrated at the fallback adapter while doing other refactoring.
Test plan:
Unit tests have been updated and `yarn test` passes. Also, I
experimentally removed the Git plugin from the list of adapters for both
the backend and frontend, and verified that the frontend UI and the
pagerank and export-graph commands still worked.
This refactors the weights module (and downstream consumers) so that
rather than tracking forwardWeight and backwardsWeight as separate
fields, we have an `EdgeWeight` type with a forwards and backwards
property. I feel this makes the code a little cleaner and wanted it a
few times while doing other refactors in this module.
Note that the graphToMarkovChain module has a distinct `EdgeWeight` type
which serves a similar purpose but happens to have different field
names. I've added a TODO to rename those fields for consistency.
Test plan: Since this is slight data structure re-organization, no new
tests are required; that `yarn test` passes is sufficient.
This pull request adds a weight slider to every NodeRow in the explorer,
enabling the user to manually set a weight for that node. The weights are
multiplicative with the type level weights, so that they can be changed
independently (e.g. you can have a comment that is weighted 2x higher than
regular comments, but still have comments get a low weight in general).
This pull coordinates a number of different changes across the codebase, all of
which are tested:
Adding support for manual weights in the weights and
weightsToEdgeEvaluator modules.
Modifying pagerankTable.TableRow so that it can show a slider in the second
column.
Adding piping for manual weights into the PagerankTable shared props, and
into the explorer app
Adding the slider to the NodeRow class that displays the current weight,
and can trigger the upstream weight change
Ensuring that the runPagerank call in the explorer actually uses the manual
weights
At present, there is no way to save these weights (they are ephemeral in the
frontend) and so this is clearly a prototype/tech demo level feature rather
than being ready for real usage. Correspondingly, CLI pagerank command always
uses an empty set of manual weights. I plan to remedy this in a follow-on pull
request.
Test plan: Run the included unit tests (yarn test) and also spin up the UI,
verify that it visually looks good in both Firefox and Chrome, and verify that
changing the weights and then re-running PageRank actually causes the cred of
the modified node to change.
Review plan: In addition to carefully reading the code, ensure that all of the
changes described a few paragraphs up are actually tested.
Merge plan: Squash and merge.
Thanks to @s-ben for proposing this feature in Discord, and to everyone
discussing its implications in this Discourse thread.
In #1140 I rename a field in PagerankGraph.ScoredNode, and in #1141 I
added a new test which referenced the field.
I merged them in quick succession, introducing a conflict. This PR fixes
it.
Test plan:
`yarn test` passes.
This commit adds a `setEdgeEvaluator` method to `PagerankGraph`,
and modifies the constructor to use that method. This will allow us to
use PagerankGraph in the explorer UI, so that we can update edge
weights without fully regenerating the graph.
Test plan: I've added new unit tests that verify basic properties of how
the edge weights are getting set and consumed.
It makes more sense that a ScoredNode has an address than that it has a
sub-node (which is an address).
Test plan: It's a simple field rename; `yarn test` suffices
Resolves#1067
Adds the CLI commands:
`sourcecred clear --all` -- removes the $SOURCECRED_DIRECTORY
`sourcecred clear --cache` -- removes the cache directory
`sourcecred clear --help` -- provides usage info
`sourcecred clear` -- prompts the user to be more specific
Test plan:
The unit tests ensure that the command is properly wired into the
sourcecred CLI, including help text integration. However, just to be
safe, we can start by verifying that calling `sourcecred` without
arguments lists the `clear` command as a valid option, and that
calling `sourcecred help clear` prints help information. (Note: it's
necessary to run `yarn backend` before testing these changes)
The unit tests also ensure that the command removes the proper
directories, so there isn't really a need to manually test it,
although the reviewer may choose to do so to be safe.
Although out of scope for unit tests on this function, we can also do
integration tests, to make sure that running the clear command doesn't
leave the sourcecred directory in an invalid state from the perspective of the `load` command.
```js
$ yarn backend;
$ node bin/sourcecred.js load sourcecred/example-github;
$ node bin/sourcecred.js clear --cache;
$ node bin/sourcecred.js load sourcecred/example-github;
$ node bin/sourcecred.js clear --all;
$ node bin/sourcecred.js load sourcecred/example-github;
```
The expected behavior of the above command block is that the load command never fails or throws an error.
@decentralion and I discussed the scenario where `rimraf` errors.
We decided that testing this scenario wasn't necessary, because
`rimraf` doesn't error if a directory doesn't exist, and
rimraf's maintainer suggests [monkey-patching the fs module]
to get rimraf to error in testing scenarios.
Thanks @decentralion for reviewing and pair-programming this with me.
[monkey-patching the fs module]: https://github.com/isaacs/rimraf/issues/31#issuecomment-29534796
This commit integrates an bare skeleton of the odyssey frontend that we
implemented in the [odyssey-hackathon] repository. You can see the
working frontend that we are trying to port over at
[sourcecred.io/odyssey-hackathon/][scio].
The prototype in the other repository has some tooling choices which are
incompatible/redundant with decisions in our codebase (sass vs
aphrodite), and requires some tools not yet present here
(svg-react-loader). This commit includes the build and integration work
needed to port the prototype frontend into mainline SourceCred. The
frontend scaffold isn't yet integrated with any "real" Odyssey data.
One potential issue: right now, every page that is rendered from the
SourceCred homepage is contained within a [homepage/Page], meaning that
it has full SourceCred website styling, along with the SourceCred
website header. The [application][scio] also has a header. Currently, I
work around this by having the Odyssey UI cover up the base header (via
absolute positioning), which works but is hacky. We can consider more
principled solutions:
- Finding a way to specify routes which aren't contained by
[homepage/Page]; maybe by adding a new top-level route
[here][route-alternative].
- Unify the headers for the Odyssey viewer and the page as a whole
(sounds like inappropriate entanglement?)
- Have a website header and also an application header (sounds ugly?)
[homepage/Page]: ee1d2fb996/src/homepage/Page.js
[route-alternative]: ee1d2fb996/src/homepage/createRoutes.js (L17)
Test plan: Run `yarn start`, and then navigate to
`localhost:8080/odyssey/`. observe that a working website is displayed,
and that the cred logo next to the word "SourceCred" is loaded properly
(i.e. svg-react-loader is integrated properly). Observe that there are
no build/compile errors from either `yarn start` or `yarn build`. Also,
observe that the UI looks passably nice, and that if the number of
elements in the entity lists is larger than can be displayed, the
sidebar pane scrolls independently.
The UI was tested in both Chrome and Firefox.
[odyssey-hackathon]: https://github.com/sourcecred/odyssey-hackathon
[scio]: https://sourcecred.io/odyssey-hackathon/
Thanks to @jmnemo, as the implementation is based on [his work].
[his work]: https://github.com/jmnemo/hackathon-event/
This commit puts in a basic data model for the Odyssey plugin. It's
built around the `OdysseyInstance` class, which is basically a Graph
that keeps around descriptions for every node, and ensures that
nodes/edges are typed in accordance with the Odyssey plugin declaration.
In the future, I want to enable instances to declare their own node/edge
types, in which case the instance will assume responsibility for
tracking and serializing the types.
To make the Odyssey plugin a 'proper plugin', I've also added a plugin
declaration, as well as analysis and explorer adapters. I haven't
decided exactly where data for Odyssey instances should be stored, so
for now the plugin adapters always return an example instance which is
based on our experience at the Odyssey hackathon.
Test plan: The instance has unit tests for its logic.
If you want to see what the plugin looks like right now when it's
integrated, you can apply the following diff, and then load the
prototype. It will contain Odyssey nodes (find them using the node type
dropdown). Note that without a seed vector to move cred back to the
values/artifacts, the cred distribution in the Odyssey subgraph is
degenerate; the users are all sinks and have postiive cred scores, but
all the other nodes converge to 0 cred.
diff --git a/src/homepage/homepageExplorer.js b/src/homepage/homepageExplorer.js
index cae4548..48987f1 100644
--- a/src/homepage/homepageExplorer.js
+++ b/src/homepage/homepageExplorer.js
@@ -6,6 +6,7 @@ import type {Assets} from "../webutil/assets";
import {StaticExplorerAdapterSet} from "../explorer/adapters/explorerAdapterSet";
import {StaticExplorerAdapter as GithubAdapter} from "../plugins/github/explorerAdapter";
import {StaticExplorerAdapter as GitAdapter} from "../plugins/git/explorerAdapter";
+import {StaticExplorerAdapter as OdysseyAdapter} from "../plugins/odyssey/explorerAdapter";
import {GithubGitGateway} from "../plugins/github/githubGitGateway";
import {AppPage} from "../explorer/App";
import type {RepoId} from "../core/repoId";
@@ -14,6 +15,7 @@ function homepageStaticAdapters(): StaticExplorerAdapterSet {
return new StaticExplorerAdapterSet([
new GithubAdapter(),
new GitAdapter(new GithubGitGateway()),
+ new OdysseyAdapter(),
]);
}
This commit modifies `PagerankGraph.runPagerank` so that the user can
provide an alpha and seed vector. The seed vector is specified via a map
of weights, which will be normalized into a probability distribution
over all the nodes in the graph. In the event that the map is empty (or
the total weight is otherwise 0), a uniform distribution is created.
To effect this change, a helper function called `weightedDistribution`
has been added (and thoroughly tested) in the `graphToMarkovChain`
module. Then, that function is used in `pagerankGraph.runPagerank`
(along with light testing).
Currently, the default alpha is set to 0, to ensure consistency with the
legacy pagerank implementation in `analysis/pagerank`. Once that has
been replaced with `PagerankGraph`, we can consider changing the defualt
alpha to non-zero (thus removing the need for synthetic self-loops).
I took a different approach in the [odyssey-hackathon repo][commit].
The previous approach was a much more complicated (and fairly redundant)
API, that allowed specifying "NO_SEED", "UNIFORM_SEED", "SELECTED_SEED",
and "SPECIFIED_SEED". I'm much happier with this API and implementation.
[commit]: ed07861073
Test plan: Unit tests included; run `yarn test`.
Summary:
the cred calculation is defined by a Markov Mixing process. By
introducing the seed vector and teleportation parameter alpha, the
Markov mixing process is augmented with a source of cred originating
from the seed vector. The resulting algorithm is the generalized
variation of PageRank, allowing computation of both canonical PageRank
where the seed vector is the uniform distribution and personalized
PageRank where the seed vector is an indicator distribution. It is still
possible to get the simple markov chain solution by setting alpha = 0.
Note that this changes the Markov process state update, but does not
provide updates to the APIs. All existing behavior is unchanged because
alpha is always set to 0.
This is a port of
https://github.com/sourcecred/odyssey-hackathon/pull/3,
which was created during the Odyssey Hackathon.
Test Plan:
Existing tests have been extended to include passing alpha = 0 to
reproduce exisiting test cases for the simple Markov Process. Additional
test cases include
- Verifying that resulting stationary distribution is unaffected by seed when alpha = 0
- Verifying that resulting stationary distribution is precisely equal to seed when alpha = 1
- Verifying that the resulting stationary distribution is linear in the seed vector
- Verifying that the correct stationary distribution is computed for non-zero alpha
- verify that the algorithm converges immediately when the initialDistribution is the stationary distribution
- verify that the changing the initialDistribution does not change the stationary distribution
Paired with @mzargham
Right now PagerankGraph requires that the user choose specific values
for maxIterations and convergenceThreshold when running PageRank.
I also rename `PagerankConvergenceOptions` to `PagerankOptions`.
The motivation is that I want to add future arguments to the same
options dict (e.g. alpha and the seed vector), so the rename is
appropriate, and allowing the options to be unset (and thus inherit
default values) will make the whole API much cleaner as I add more
options.
Test plan: Unit test added. `yarn test` passes.
In [#1128: Add support for seed vectors][#1128], we significantly
increase the number of arguments to
markovChain.findStationaryDistribution. To clean up the invocations, I
added a followon PR (#1129) which converts findStationaryDistribution to
use a `PagerankParams` object instead.
However, I think it will be cleaner to land the PagerankParams refactor
before adding new features in #1128, so I'm making this PR as
pre-cleanup.
Test plan: This is a trivial refactor. `yarn test` passes.
[#1128]: https://github.com/sourcecred/sourcecred/pull/1128
This is a minor refactor so that we use Aphrodite for styling on
HomePage.js. It's not super consequential, but I want to switch to using
Aphrodite more consistently in the codebase, so why not start here.
Test plan:
`yarn test` reveals no errors.
`yarn start` launches a correctly styled frontend.
I also used `build_static_site.sh` and the resultant site is also
correctly styled.
This commit updates the `sourcecred load` command so that it also
automatically runs PageRank on completion.
The implementation is slightly hacky, in that it prints two sets of
task status headers/footers to console, for reasons described in a
comment in the source code. The user-visible effect of this hack can
be seen below:
```
❯ node bin/sourcecred.js load sourcecred/example-github
Starting tasks
GO load-git
GO load-github
DONE load-github
DONE load-git
Overview
Final result: SUCCESS
Starting tasks
GO run-pagerank
DONE run-pagerank
Overview
Final result: SUCCESS
```
It would be good to clean this up, but for now I think it's acceptable.
Note that it is not safe to assume that a PagerankGraph always exists
for repos that are included in the RepoIdRegistry. The repo gets added
to the registry before the pagerank task runs. Consumers that are
loading the `PagerankGraph` can just check that the file exists, though.
Test plan: I've added unit tests that verify that the right tasks are
generated. Most importantly, the snapshot of the results of `sourcecred
load` now include a snapshotted pagerank graph.
(The snapshot was updated via `UPDATE_SNAPSHOT=1 yarn test --full`.)
Further progress on #967.
Across SourceCred usage, we depend on the `SOURCECRED_GITHUB_TOKEN`
environment variable being set. Confusingly, some tests expect
`GITHUB_TOKEN` instead of `SOURCECRED_GITHUB_TOKEN`.
This commit resolves that inconsistency, by having all tests that read
from the environment use `SOURCECRED_GITHUB_TOKEN`. This was already
available as a secret in our CI configuration, so no change is needed
there. (After this merges, we may remove the GITHUB_TOKEN variable from
the environment.)
Test plan: `yarn test --full` passes without the environment variable
set. Also, the following grep produces only innocuous hits.
```
git grep -P "(?<\!SOURCECRED_)GITHUB_TOKEN"
```
This commit refactors the `sourcecred load` CLI command so that it uses
dependency injection, much like the testing setup #1110. This makes it
feasible to test "surface logic" of how the CLI parses flags and
transforms them into data separately from the "piping logic" of invoking
the right API calls using that data.
This is motivated by the fact that I have other pulls on the way that
modify the `load` command (e.g. #1115) and testing them within the
current framework is onerous.
Test plan:
This is a pure refactoring commit, which substantially re-writes the
unit tests. The new unit tests pass (`yarn test --full` is happy).
Note that `yarn test -full` also includes a sharness test that does an
E2E usage of `sourcecred load`
(see sharness/test_load_example_github.t), so we may be confident that
the command still works as intended.
This commit fixes three broken links (two in the README, one in the prototype app) that were still pointing to https://discuss.sourcecred.io/.
Test plan:
Verify that there are no other bad links to the old Discourse location, by running `git grep "discuss.sourcecred.io"`.
This commit adds a new CLI command, `pagerank`, which runs PageRank on a
given repository. At present, the command only ever uses the default
weights, although I plan to make this configurable in the future. The
command then saves the resultant pagerank graph in the SourceCred
directory.
On its own, this command is not yet very compelling, as it doesn't
present any easily-consumed information (e.g. users' scores). However,
it is the first step for building other commands which do just that. My
intention is to make running this command the last step of `sourcecred
load`, so that future commands may assume the existence of pagerank
scores for any loaded repository.
Test plan: The new command is thoroughly tested; see
`cli/pagerank.test.js`. It also has nearly perfect code coverage (one
line missing, the dependency-injected real function for loading graphs).
Additionally, the following sequence of commands works:
```
$ yarn backend
$ node bin/sourcecred.js load sourcecred/pm
$ node bin/sourcecred.js pagerank sourcecred/pm
$ cat $SOURCECRED_DIRECTORY/data/sourcecred/pm/pagerankGraph.json
```
Material progress on #967.
This commit adds a module, `fetchGithubOrg`, which loads data on GitHub
organizations, most notably including the list of repositories in that
org.
The structure of this commit is heavily influenced by review feedback
from @wchargin's [review] of a related PR.
Test plan: This logic depends on actually hitting GitHub's API, so the
tests are modeled off the related `fetchGithubRepo` module. There is a
new shell test, `src/plugins/github/fetchGithubOrgTest.sh`, which
verifies that that the org loading logic works via a snapshot.
To verify the correctness of this commit, I've performed the following
checks:
- `yarn test --full` passes
- inspection of `src/plugins/github/example/example-organization.json`
confirms that the list of repositories matches the repos for the
"sourcecred-test-organization" organization
- manually breaking the snapshot (by removing a repo from the snapshot)
causes `yarn test --full` to fail
- running `src/plugins/github/fetchGithubOrgTest.sh -u` restores the
snapshot, afterwhich `yarn test --full` passes again.
[review]: https://github.com/sourcecred/sourcecred/pull/1089#pullrequestreview-204577637
This pulls the logic for loading a SourceCred graph from disk out
`cli/exportGraph` and into `analysis/loadGraph`. The rationale is that
`exportGraph` is not the only command that wants the ability to load a
graph from the analysis adapters.
The new command has a clean return signature that reports whether the
load was successful, or failed because the graph wasn't loaded, or
failed due to an error in plugin code.
Testing of the loading logic has been moved to `loadGraph.test`, and the
CLI has been refactored so that the loadGraph method is dependency
injected. This allows for (IMO) cleaner testing of the CLI method.
There is one (deliberate) change in behavior, which is that the command no
longer throws an error if no plugins are included; instead it will just
export an empty graph. I don't have a strong preference between the two
behaviors; changing it was just more convenient.
Test plan: New unit tests have been added, and tests of the cli command
have been re-written. As a sanity check, I've verified that the
following sequence still works:
```
$ yarn backend
$ node bin/sourcecred.js load sourcecred/pm
$ node bin/sourcecred.js export-graph sourcecred/pm
```
Nearly perfect code coverage is maintained. One line is uncovered, and
it's the line that injects in the actual graph loading behavior.
This commit adds a `neighbors` method to `PagerankGraph`. This is an
augmented version of `Graph.neighbors`. It returns the base data from
`Graph.neighbors` as well as the score, the edge weights, and the score
contribution. The score contribution basically means how much score was
contributed from the target node by this particular neighbor connection.
When the graph is well-converged, a node's score will be the sum of all
its neighbors' score contributions, as well as the contribution it
received from its synthetic loop edge. So, for completeness sake, I
added another method, `syntheticLoopScoreContribution`, which computes
how much score a node received from its synthetic loop edge. (This value
should usually be close to 0).
You can think of these two methods as providing a replacement for the
`PagerankNodeDecomposition` logic.
Test plan: I've added tests that verify:
- That neighbors returns results consistent with Graph.neighbors
- That neighbors' score contributions are computed correctly
- That neighbors errors if the graph has been modified
- That synthetic score contributions are computed correctly
- That a node's score is the sum of all its contributions
Test plan: Unit tests included. Run `yarn test`.
This commit moves the default Pagerank options out of
`analysis/pagerank` and to `core/pagerankGraph`. This reflects the
gradual migration of core pagerank logic into `pagerankGraph`.
Test plan: `yarn test` should suffice. It's a trivial change.
* Show tooltips in weightConfig UI
* Updated to pass checks from prettier
* Updates unit tests to check WeightSlider descriptions
* Update CHANGELOG.md to reflect PR #1081
* Add CLI command: `sourcecred export-graph`
This adds an `export-graph` command to the SourceCred CLI. It exports
the combined cred graphs for individual RepoIds, as was done for
[sourcecred/research#4].
Example usage:
```
$ node bin/sourcecred.js load sourcecred/mission
$ node bin/sourcecred.js export-graph sourcecred/mission >
/tmp/mission_graph.json
```
Test plan:
The new command is thoroughly unit tested.
[sourcecred/research#4]: https://github.com/sourcecred/research/pull/4
* Address review feedback by @wchargin
This commit makes several improvements to `repoIdRegistry`:
- Create `writeRegistry` and `getRegistry` methods to abstract over
needing to find the right file, populate an empty registry if its not
available, etc.
- Create `getEntry` for efficiently checking whether a RepoId is in the
registry
- Rename `addRepoId` to `addEntry` for consistency
- Add docstrings
The `load` command has been refactored to use the new methods.
Test plan: Unit tests added, and they pass. The `load` command is
already thoroughly tested, so regressions are very unlikely.
Part of ongoing work for #1020.
Test plan:
Added tests that mirror the edge filtering tests in `graph.test`
to check that `graph` and `pagerankGraph` return the same edges
with the given `EdgesOptions` parameter. Also added a sanity check
that a `weight` prop is returned from the iterator along with the edge.
Given the dependence on a helper function to test the edge
iterator's equality between graphs, I would suggest reviewers give
particular attention to that function:
`expectConsistentEdges()`
This commit adds a `totalOutWeight` method to `PagerankGraph`.
For any given node, `totalOutWeight` reports the total weight traveling
away from the node on edges (including the synthetic loop edge). Using
totalOutWeight makes it possible to normalize the weights to get the
actual markov transition probabilities.
Test plan: Unit tests verify the following properties:
- An error is thrown if the requested node does not exist.
- An error is thrown if the graph has been modified.
- The out weights are computed correctly in the standard case.
- The out weights are computed correctly in the case where there are no
weights (except the synthetic loop weight)
- The out weights are still computed correctly after
JSON-deserialization.
Inspired by a [suggestion] @decentralion made to improve #1105
This will enable `pagerankGraph` to throw an error when it is
called with invalid option parameters. Previously, to elicit
this error we had to access the iterator through `Array.from()`
or similar.
Test plan:
Yarn test passes.
Specifically, I removed the `Array.from()` wrapper around `pagerankGraph`
in the test that checks to see that `pagerankGraph` throws an error when
`nodes()` is passed invalid options.
[suggestion]: https://github.com/sourcecred/sourcecred/pull/1105#pullrequestreview-206496537
The motivation for this change is to make it easier
to access the selected Node's `name` prop for #576,
in which we plan to show a Card displaying summary
stats for the selected node. With only the `topLevelFilter`
available, it's trickier than it needs to be to find out
a node type's `name`.
Test Plan:
* Yarn test passes.
* Visual/Manual inspection of table doesn't surface any issues.
* Updated `it("filter defaults to defaultNodeFilter if available")`
to `it("selectedNodeType defaults to defaultNodeType if available")`.
* Verified that the above new test is failable in several ways by
mangling the tests to test for the wrong node type and mangling the
code to set the wrong node type.
* Since we factored out 'topLevelFilter' and 'defaultNodeFilter',
running `git grep -i topLevelFilter` and `git grep -i defaultNodeFilter`
turns up empty, just to make sure those terms aren't hanging
around to confuse anybody in the future.
* I don't think changing the `prop` parameter warrants any
additional tests, as the current tests verify that the prop
is passed in correctly.
This was at @decentralion's suggestion, following the Contributing
Guideline's Kent Beck quote of making the easy change to make the
change we were originally after (#576) easier. 🙌
* Highlight tableRows on :hover and :focus-within
Resolves#1041
The purpose of this commit is to make the pagerankTable easier
to read, as it's currently difficult to distinguish which score is
associated with which row because of the tight spacing of the
rows and the space between the score column and the row detail column.
@wchargin provided the implementation using `linearGradient()`
and `backgroundImage`s.
A side effect of highlighting the row on `focus-within` is that the rows
will become highlighted when the expand button is clicked, which we
decided was acceptable.
Test plan:
Yarn test passes.
To test the new highlight behavior, visual/manual inspection
passes.
Also added the Aphrodite className to the snapshot
tests. The combination of testing the className + inline style props
should make these tests sensitive to UI changes in the future.
Screenshots:
<img width="939" alt="screenshot 2019-02-17 15 46 34" src="https://user-images.githubusercontent.com/26695477/52918955-332f5280-32cb-11e9-87d3-887c8877116a.png">
<img width="931" alt="screenshot 2019-02-17 15 45 10" src="https://user-images.githubusercontent.com/26695477/52918953-2f9bcb80-32cb-11e9-9356-82c6dccab4ae.png">
* bump CI
Suggested by @decentralion in his review of #1090
Test plan:
yarn test passes. Also verified that the new test case is
failable, if you pass in the wrong array of nodes to `expect()` or
if you mangle the node filter code.
Continuing work on #1020.
Adding an optional parameter to `nodes()` which enables optional
node prefix filtering.
Test plan:
@decentralion suggested on Discord that the tests should verify:
1) the parameter was passed to `_graph` correctly
2) the augmentation logic was applied correctly
The tests I added are identical to the tests in `graph.test`, except
that they verify that the result of `pagerankGraph` matches that of
`graph`. On one hand, this creates a dependence on `graph`,
as these tests don't verify that the filter works correctly, only that
graph has applied the filter and returned the iterator.
However, my prevailing thought is that it isn't `pagerankGraph's` responsibility
to test the behavior of `graph`, and so testing the exact filter results
of `pagerankGraph` like we do in `graph.test` isn't the best strategy, and
testing that `pagerankGraph`'s results equal `graph`'s results is a better strategy.
The tests also check that a `score` is provided alongside each `node` in the iterator,
to minimally satisfy @decentralion's second spec.
yarn test passes.
* PagerankGraph: Add toJSON/fromJSON
This commit adds serialization logic to `PagerankGraph`. As with many
things in PagerankGraph, it's based on the corresponding logic in `Graph`.
Much like graph, it stores data associated with nodes and edges (in this
case, the scores and edge weights) in an ordered array rather than a
map, so as to avoid repetitiously serializing the node and edge
addresses.
Test plan: Unit tests added, and they should be sufficient. Also take a
look at the included snapshot.
Part of ongoing work for #1020.
Adds an equals method for the PagerankGraph. This is really quite
straightforward, the logic is based on the matching logic for
`Graph.equals`.
Tests added.
Test plan: The added tests are comprehensive, and they pass.
As discussed in #1004, we want to be able to package metadata with a
graph's nodes and edges. We can do this much more compactly if we store
the metadata as an array, ordered by the corresponding node/edge
address, rather than storing a map. The disadvantage is that clients
then need to manually sort the graph addresses to deserialize.
This commit adds public methods that allow a client to efficiently
retrieve the sorted addresses from the GraphJSON (where they are already
serialized). This behavior is tested. Note that we appropriately don't
allow clients to peek and directly depend on the exact representation of
GraphJSON, we just promise that sorted address retrieval is possible.
Test plan: Unit tests added (and I verified that breaking the sorting
will fail the test).
* Start work on the PagerankGraph
This commit begins work on the `PagerankGraph` class, as described in
[#1020]. As of this commit, the `PagerankGraph` has basic functionality
like retrieving nodes and edges, and running PageRank. However, it is
missing utility functionality like equality testing and serialization,
and doesn't yet have score decomposition logic.
This was mostly produced during a [live coding session]. Thanks to
@BrianLitwin, @anthrocypher, and @wchargin for participating.
Test plan:
The new code is thoroughly unit tested. Please review the test coverage,
and also the quality of the documentation.
[#1020]: https://github.com/sourcecred/sourcecred/issues/1020
[live coding session]: https://github.com/sourcecred/mission/issues/14
* Improvements from self-review
- Don't allow PRG around empty graph, as there's no way to make it
a valid probability distribution
* Add issue ref in TODOs
This commit modifies `markovChain.findStationaryDistribution` so that
in addition to returning the final distribution, it also reports the
final convergence delta.
This is motivated by the proposed API for the new PagerankGraph (see
[#1020]). Also, I think it makes a nice addition to the test code.
Note that this slightly changes the output from `findStationaryDistribution`,
because we now return the first distribution that is sufficiently converged,
rather than that distribution with one additional Markov action.
Test plan:
Unit tests are updated, and `yarn test` passes.
[#1020]: https://github.com/sourcecred/sourcecred/issues/1020
Thanks to @BrianLitwin for semi-pair-programming it
Thanks to @wchargin for extensive review feedback.
Pull #1080 added in a description field for edge types, but put in a
placeholder message for each actual description. This pull adds in
descriptions for each edge type.
Test plan: `yarn test` passes, and additionally
`git grep 'TODO: Add a description for this edge type'` returns no hits.
Reviewed by @BrianLitwin and @wchargin.
* Enable loading private git repositories
This commit enables loading private repositories, assuming that the user
has ssh-agent configured with keys to allow cloning the private
repository, and has provided a GitHub API token with permissions for the
repository in question.
I have not added automated testing. I don't think a cost-benefit
analysis favors adding such tests at this time:
- This code changes very infrequently, and so is unlikely to break
- If it does break, it will be pretty easy to catch and to fix
- the @sourcecred org is on a free plan, which doesn't allow private
repos, so setting up the test case is a bit of a pain
Test plan: `yarn test --full` passes, so I haven't broken existing Git
clone behavior. Locally, I am able to load private repositories.
* Remove unnecessary process import.
Fixes#1019.
Test plan: Loading the prototype works, as does clicking through to different prototype pages.
“Running `git grep -F '/prototypes/'` returns no results; before this commit, it yielded 2 results.”
This commit #811, allowing users to set the weights of node/edge types to 0.
The WeightSlider now sets the weight to 0 when its dragged to its minimum value.
The logic for converting between weights and sliders has also been made more robust,
and is more thoroughly tested.
In cases where we wanted to set the weight to 0 (e.g. backwards Reaction edges),
the default weight has been changed.
Test plan:
Loading the UI, check that the sliders still work as expected (dragging them changes the displayed weight, dragging to the far left sets weight to 0). Check that the weights are consumed as expected (setting weight for issues to 0 leads to no cred for issues). Check that the weights for backwards reaction edges now have 0 weight. `git grep "TODO(#811)"` returns no hits.
PR #1075 added a new EdgeType, and #1080 added a new field to EdgeTypes.
Both PRs merged and this broke the build.
This very trivial commit fixes the build breakage in a noncontroversial
way (copies the placeholder edge description used for every other edge
over).
Test plan: `yarn test` passes.
Updating github example data with support
for 🚀 and 👀 reaction types.
This follows #1068 and @decentralion updating
the archived repo with the new reaction types.
`src/plugins/github/fetchGithubRepoTest.sh -u`
(as @decentralion suggested) updated `example-github.json`
`yarn unit` caught two tests with failing snapshot
tests (`createGraph.test` and `relationalView.test`), so
I updated those with `yarn unit -u`
`yarn test -full` caught a failing snapshot test
at `sharness-full`, resolved by updating the
snapshot in `view.json.gz` with
`UPDATE_SNAPSHOT=1 yarn test --full`.
Thanks to @wchargin for the [explanation] on how
to resolve that issue.
[explanation]: https://github.com/sourcecred/sourcecred/pull/1077#pullrequestreview-196805017
**Test Plan:**
`yarn test --full` is passing.
Additionally, the commands:
```sh
filepath="./sharness/__snapshots__/example-github-load/data/sourcecred/example-github/github/view.json.gz" &&
[ -f "${filepath}" ] && # sanity check
diff -u \
<(git show "HEAD:${filepath}" | gzip -d | jq .) \
<(gzip -dc "${filepath}" | jq .) \
;
```
yields the following output:
```
--- /dev/fd/63 2019-01-27 08:34:15.020387301 -0500
+++ /dev/fd/62 2019-01-27 08:34:15.021095696 -0500
@@ -654,6 +654,22 @@
"subtype": "USER",
"login": "decentralion"
}
+ },
+ {
+ "content": "ROCKET",
+ "user": {
+ "type": "USERLIKE",
+ "subtype": "USER",
+ "login": "decentralion"
+ }
+ },
+ {
+ "content": "EYES",
+ "user": {
+ "type": "USERLIKE",
+ "subtype": "USER",
+ "login": "decentralion"
+ }
}
]
}
```
Again, thanks @wchargin's for providing those commands and accompanying
explanation.
Resolves#1054
Added "ROCKET" and "EYES" to the list of reaction types.
Added "ROCKET" as a valid cred signal, kept "EYES" invisible.
My approach was to use `git git grep THUMBS_UP '*.js'`
and `git grep ThumbsUp '*.js'` to find all the relevant files,
as suggested in #1054
**Test Plan**
1) Inspecting Sourcecred/Mission's UI:
[#13] contains: GOT 🚀 FROM 1 user
@BrianLitwin contains: REACTED 🚀 TO 1 issue
@BrianLitwin contains: REACTED 🚀 TO #13
2) Yarn Test passes
3) `github/edges.test` includes a snapshot test to verify
that we can create an edge using ROCKET
4) @wchargin also noted that:
```sh
diff -u <(git grep -c 'THUMBS_UP' '*.js') <(git grep -c 'ROCKET' '*.js')
diff -u <(git grep -c 'ThumbsUp' '*.js') <(git grep -c 'Rocket' '*.js')
```
passes.
`graphql/mirror.test` now includes "ROCKET" and "EYES" in the example
GithubSchema, but their inclusion has no effect
on any tests.
**Screenshots**
1.
<img width="378" alt="screenshot 2019-01-22 09 02 12" src="https://user-images.githubusercontent.com/26695477/51540428-6c87b600-1e24-11e9-8334-1d9d993dce01.png">
2.
<img width="525" alt="screenshot 2019-01-22 09 02 41" src="https://user-images.githubusercontent.com/26695477/51540472-84f7d080-1e24-11e9-8847-245c0c09ddd6.png">
<br>
Shoutout to [this comment], which saved me an untold amount of head-scratching,
and also @Decentralion's help debugging in the Issue thread.
[#13]: https://github.com/sourcecred/mission/issues/13
[this comment]: e0762303d4/src/plugins/github/graphqlTypes.test.js (L13-L15)
* Add descriptions for NodeTypes
As highlighted by @decentralion in issue #807, we need descriptions for Node and
Edge types in the UI to explain to users what each Node and Edge type does. This
PR modifies the type definition for `NodeType` and adds a `+description: string`
field, then updates all NodeTypes throughout the codebase with descriptions.
Test plan:
Verify that all tests pass and the descriptions makes sense.
This commit adds a new `modificationCount` method to `Graph`, which
exposes's that graph's modification count. This enables clients to write
cached data structures on top of Graph, knowing that they can
programatically detect when the cache has been invalidated.
Test plan: Unit tests have been addded; `yarn test` passes.
This commit is motivated by work on #1020.
There are two kinds of plugin adapters: adapters for doing cred
analysis, called "analysis adapters", and adapters for the cred
explorer, which are confusingly called "app adapters".
This commit decreases the confusion by renaming app adapters to explorer
adapters across the codebase. In a future commit, I will add
documentation to the adapter interfaces so that it is clearer to a
newcomer to the codebase why these interfaces exist.
Thanks to @BrianLitwin, who asked a question during [office hours]
that surfaced this issue.
[office hours]: https://github.com/sourcecred/mission/issues/12
Test plan: `yarn test` passes, suggests that this rename went off
without a hitch. Code review as a sanity check.
Also: grepping for `AppAdapter` returns 0 results:
```
$ git grep AppAdapter | wc -l
0
```
Note: After producing this commit, I can confirm that the word "adapter"
starts to look like utter gibberish after you type it often enough.
This commit adds some docstrings for the concepts of NodeType and
EdgeType. I also swapped the order so that `NodeType` comes first,
which is more consistent with usage across the codebase.
This commit makes no changes to the actual code; the only effects
are re-organization and documentation.
Test plan: `yarn test` && human inspection
Summary:
We use Aphrodite, not CSS imports, for styling. We do have a small
`index.css` file that is included during server-side rendering, and is
only referenced from `src/homepage/server.js`. But our `index.js` file
also has a superfluous `import "./style.css"`, which might suggest that
we support CSS imports more generally. This patch removes that import.
Thanks to @brianlitwin on Discord for pointing out that this might be
confusing.
Test Plan:
Verified that, under both `yarn start` and `yarn build`, the appearance
is the same, and the document still includes a `<style>` element with
the contents of `index.css` (which is included by `server.js`).
wchargin-branch: remove-css-import
* Add documentation to the Graph module
This commit adds a module-level docstring that gives an overview of the
Graph class and its importance to SourceCred, as well as adding
docstrings to specific methods.
Test plan:
In addition to review by the SourceCred maintainers, this should be
reviewed by at least one person who is not familiar with the codebase,
so that we can verify that it's actually working as documentation. :)
* Incorporate @wchargin's many suggestions.
Test plan: Human review.
This commit adds a new runOption for execDependencyGraph, namely
`printVerboseResults`. If this flag is true, then execDependencyGraph
will print a "Full Results" section along with the standard error and
standard out of every task, regardless of whether it failed or
succeeded. (Note, this is the existing behavior for all invocations
prior to this commit).
If the flag is not true, then execDependencyGraph will not print a full
results section, and stdout/stderr will be logged only for tasks that
fail.
This commit also modifies `yarn test` to use the new flag so that it
prints verbose tests only when the `--full` option is provided. This is
consistent with our sharness behavior: we print the full sharness logs
only when `--full` was provided.
This fixes#1035, and ensures that running `yarn test` has a high signal
to noise ratio (i.e. it only shows an enumeration of top level tasks).
This improves the developer ergonomics of SourceCred by not having a
super commonly used and core script spam the user with mostly irrelevant
information.
Test plan:
Run `yarn test` when all tests are passing, and observe that the output
has much less noise:
```
yarn run v1.12.3
$ node ./config/test.js
tmpdir for backend output: /tmp/sourcecred-test-6337SZ9smvWsWvqE
Starting tasks
GO ensure-flow-typing
GO check-stopships
GO check-pretty
GO lint
GO flow
GO unit
GO backend
PASS check-stopships
PASS ensure-flow-typing
PASS flow
PASS backend
GO sharness
PASS sharness
PASS check-pretty
PASS lint
PASS unit
Overview
Final result: SUCCESS
Done in 11.66s.
```
Run `yarn test` when there is a real failure (e.g. a unit test failure)
and observe that full details on the failure, including the output from
stdout/stderr, is still provided.
Run `yarn test --full` and observe that full, verbose logs are provided.
Resolves#1027
Using `repoId.owner/repoId.name` for the project title
because that is how projects are identified on `PrototypePage`.
Created a `<ProjectDetail />` component inside `<App />` that consumes a `RepoId`
and renders a title.
**Test Plan:**
Added two unit tests:
The first verifies that the parent `<App />` component
instantiates a `<ProjectDetai />` component with the correct props.
The current correct prop is a `RepoId` object.
The second test verifies that the `<ProjectDetail />` component renders
the title correctly given the `RepoId`, ie as a `<p>` element
with `repoId.owner/repoId.name` for text.
Visual tests verify that the title is above the Analyze Cred
button, and that clicking from one project to another renders
the appropriate title for separate projects.
Attaching a screenshot as a comment at #1032
for reference:
<img width="1253" alt="screenshot 2019-01-04 13 40 03" src="https://user-images.githubusercontent.com/26695477/50706562-34aeff00-102c-11e9-9c1c-6c1e3fa6c415.png">
This moves the invariant checking code from the top of the Graph class
to the bottom. Most readers of this file will probably be more
interested in seeing the API, and reading the invariant checker first
is likely to be confusing and off-putting.
Test plan: `yarn test` suffices. No semantic change.
This commit substantially improves SourceCred's performance in
production.
Measurement methodology: I create a new tab in Chrome, navigate to my local
prototypes, and select go-ipfs. I then turn on profiling, and click the
analyze button, and then turn off profiling when analysis is done. I
then go to the "bottom-up" tab in the JS analysis box on the bottom and
sort by "Total Time".
__Before this commit:__
| fn | total time | time as % |
|:---------------- | ----------:| ---------:|
| assertValid | 815ms | 8.6% |
| assertValidParts | 261ms | 2.7% |
__After this commit:__
| fn | total time | time as % |
|:---------------- | ----------:| ---------:|
| assertValid | 21ms | 0.2% |
| assertValidParts | 23ms | 0.3% |
Test plan: `yarn test`, also performance measurement as described above.
Fixes#1011.
This adds a blacklisted id for @greenkeeper, a bot which used to be a
user. This is a temporary fix until we solve #998.
Test plan: `yarn test` passes. Before this commit, attempting to load
`probot/probot` fails. After this commit, it succeeds.
When I implemented this function, I incorrectly assumed that
`lodash.sortBy` only calls subsequent accessor functions if there is a
tie from the first accessor. Actually, it calls it every time. We can
avoid lots of wasteful JSON.serialization by just grabbing the exact
properties of interest.
Test plan:
For correctness: `yarn test` suffices, as this functionality is already
tested.
For performance improvement: I ran the full load+analyze workflow, in
Chrome, on twbs/bootstrap. Before this change, decompose took 6.9s;
after this change, it takes 1.3s, for a 5.3x speedup.
Close#943.
Summary:
Our registry was defined to simply be a list of IDs. This is
insufficiently flexible; we want to be able to annotate these IDs with,
e.g., last-updated times (#989). This commit wraps the entries in a
simple object, updating clients appropriately.
Test Plan:
- Run `node ./bin/sourcecred.js load sourcecred/example-github` with a
repository registry in the old format, and note that it errors
appropriately.
- Run `yarn build` with a repository registry in the old format, and
note that it errors (“Compat mismatch”).
- Delete the old registry and re-run the `load` command. Note that it
runs successfully and outputs a registry. Run `yarn build`; note
that this works.
- Load data for two repositories. Run `yarn start`. Note that the list
of prototypes still works, and that you can navigate to and render
attributions for individual project pages.
- Verify that `yarn test --full` passes.
wchargin-branch: repo-id-registry-metadata
Summary:
There have been some breaking changes that require new type annotations,
which is a good thing: these prevent `any`-leakage.
Test Plan:
Run `yarn flow`.
wchargin-branch: flow-v0.86.0
Summary:
This serves as a regression test for #1000.
Test Plan:
Note that `yarn unit` passes with this patch but fails if the change to
the code is reverted, or if the patch in #1000 is reverted. Note that
`yarn build` also passes but fails if the patch in #1000 is reverted.
Note also that `yarn test --full` passes.
wchargin-branch: link-verify-trailing-slash
Summary:
Prior to this commit, clicking the in-copy link to the prototypes page
would raise a console error:
> Warning: [react-router] Location "/prototype" did not match any routes
Test Plan:
Run `yarn start` and click the link.
wchargin-branch: site-fix-homepage-prototype-link
Summary:
Prior to this commit, the prototypes page, which lists just a handful of
repositories, was rendered with a vertical scrollbar: you had to scroll
200px to see the version info. This is silly.
The `height: 100%` is necessary not to get it to fill up the whole page,
but to get it to _not_ fill up ~30 extra pixels. I have no idea why.
Test Plan:
Run `yarn start` and note that `/prototypes/` now renders without a
scrollbar, and with the version info in the bottom-right corner.
wchargin-branch: site-fix-prototypes-page-dimensions
This resolves an outstanding TODO in pagerankNodeDecomposition to remove
the unused sourceScore field.
I have removed it, and it was indeed unused.
Test plan: `yarn test` passes.
Summary:
The `MapUtil` map–object conversion functions used inexact objects for
both input and output. They are in fact stronger than that: they can
accept arbitrary inexact objects and return arbitrary exact outputs.
(Recall that exact objects are subtypes of their inexact counterparts,
so this is the maximally permissive combination.)
Test Plan:
Unit tests added. The “can return an exact object” test fails Flow
before this change. The other tests would have passed already.
wchargin-branch: maputil-exact-output
RepoIdRegistry is used across the project, but not in the explorer. So
it makes very little sense that it live in the explorer module. It's now
moved to core.
Test plan: `yarn test --full` passes
We added a configurable cred feedback url on the theory that we would
create a separate discourse post to collect feedback for each project in
particular.
We've now realized that no one is using this, so it's just vestigial
complexity now. So I'm removing the logic for configuring the feedback
url on a per-project basis.
Instead, we will always link to a Google form for collecting feedback.
Test plan: `yarn test --full` passes, and I manually checked the links.
Historically, a single cred explorer instance could load many different
repositories. This turned out to be an anti-feature: we'd rather have a
particular url hardlink to exploring the cred for a particular project.
This commit removes the repository select from the explorer, and instead
mandates that the explorer always has the RepoId passed down from above.
Besides providing a better UX, this also greatly simplifies the logic
for the explorer, since we no longer have an "initializing state" that
doesn't have any RepoId.
This builds on the work in #984, and swaps out the old "prototype" page
(which has been rendered non-functional by this change) for the new
"prototypes" page. Note that it stays at the same route, so links to
sourcecred.io/prototype will continue to function.
Test plan: Ran `yarn test --full`, and verified that `yarn start`
produces a working site.
Test Plan:
Apply the following patch:
```diff
diff --git a/src/homepage/routeData.js b/src/homepage/routeData.js
index 32d3eb65..aac7fc9a 100644
--- a/src/homepage/routeData.js
+++ b/src/homepage/routeData.js
@@ -38,7 +38,10 @@ const routeData /*: $ReadOnlyArray<RouteDatum> */ = [
path: "/prototypes/",
contents: {
type: "PAGE",
- component: () => require("./PrototypesPage").default([]),
+ component: () =>
+ require("./PrototypesPage").default([
+ {owner: "sourcecred", name: "example-github"},
+ ]),
},
title: "SourceCred prototypes",
navTitle: null, // for now
```
Then, load <http://localhost:8080/prototypes/> and see that there is an
entry in the list, and that it links to
<http://localhost:8080/prototypes/sourcecred/example-github/>. Note that
clicking the link raises a console error because there is no such route.
wchargin-branch: homepage-prototypes-page
Summary:
We want to remove the repository selector dropdown on the cred explorer
homepage and instead render a separate web page for each project. To do
this, we need to know which pages to render statically. We choose to
ingest this information from the state of the repository registry at
build time.
This commit adds an environment variable `REPO_REGISTRY` whose contents
are the stringified version of the repository registry, or `null` if
SourceCred has been built for the backend. This variable is defined with
Webpack’s `DefinePlugin`, so any code bundled by Webpack can refer to it
via `process.env.REPO_REGISTRY` both on the server and in the browser.
Paired with @wchargin.
Test Plan:
Sharness tests modified; running `yarn test --full` suffices.
The explorer no longer ships with a set of default plugins. (This made
an inappropriate dependency from explorer/ to plugins/, and complicated
explorer's contract as a generic component.) Instead, the homepage
module is responsible for choosing the plugins to display on the
homepage.
Test plan: `yarn test --full` passes, and `yarn start` reveals a
functioning homepage and prototype.
Currently version is located in `homepage/`, which doesn't make much
sense, since it's versioning the whole project.
We move it to core.
Test plan: `yarn test --full`
Summary:
This was used for ad hoc testing of the Mirror module before it was
integrated into SourceCred. We haven’t kept it up to date with schema
changes, and it is no longer needed: you can just run `sourcecred load`.
This was also the only untested code in the `graphql/` package, so it is
nice to remove it.
Test Plan:
Running `yarn test --full` passes.
wchargin-branch: remove-mirror-demo
Summary:
This adds object IDs to the GitHub GraphQL blacklist such that the
`twbs/bootstrap` repository can be loaded.
Ingesting the Mirror-extracted data into the RelationalView yields the
warnings
```
IssueComment[MDEyOklzc3VlQ29tbWVudDEwNTI4Mzk4Ng==].reactions: unexpected null value
IssueComment[MDEyOklzc3VlQ29tbWVudDI0NTQ3OTM3OA==].reactions: unexpected null value
IssueComment[MDEyOklzc3VlQ29tbWVudDMwNDE4NzIzMg==].reactions: unexpected null value
```
because we have nulled out these `Reaction`s in their enclosing
connections. This is expected.
Test Plan:
Run `yarn backend` and `node ./bin/sourcecred.js load twbs/bootstrap`.
Run `yarn start` and note that the cred attribution renders properly.
(Loading the GitHub data may take an hour or two. The resulting SQLite3
database is 172MB. Ingesting it into the `RelationalView` still takes
just a few seconds, and the cred attribution is rendered quickly.)
wchargin-branch: github-use-blacklists
Summary:
This enables us to deal with GraphQL remotes that violate their contract
guarantees and provide a node of the wrong type. An instance in which
the GitHub GraphQL API does this is documented here:
<https://gist.github.com/wchargin/a2b8561b81bcc932c84e493d2485ea8a>
A Mirror cache is only valid for a fixed set of blacklisted IDs. This is
necessary to ensure consistency. If the set of blacklisted IDs changes,
simply remove the existing database and download again from scratch.
Test Plan:
Unit tests added, with full coverage.
wchargin-branch: mirror-blacklist
Currently, the cred explorer is a submodule of `app`. This is somewhat
confusing, as `app` is essentially our homepage, and the explorer is a
standalone React application which happens to get embedded in our
homepage. This commit pulls the explorer from `app/credExplorer/` into
`explorer/`, which is a better organization.
The `app/adapters` were actually only used by the cred explorer, so
those files have been moved to `explorer/adapters`. We should rename
them from "App Adapters" to "Explorer Adapters", but I didn't do that in
this commit so as to minimize the (already substantial) size of the
change.
Also, we should rename `app/` to `homepage/` in a subsequent commit.
I encountered a nasty Flow bug, which I fixed with help from @wchargin.
The result is extra annotations on the demo and fallback dynamic
adapters (so that the `static()` method is type annotated).
Test plan: This change is massive, but it's just a rename. `yarn test`
suffices.
I'm planning to pull `credExplorer` out of `app` and into its own
top-level module. This is a bit awkward, as `credExplorer` depends on
a lot of little modules that are currently collected in `app/`.
To resolve this, I pull all of these little utility modules into
`webutil/`. It's not a totally principled grouping, but it's quite
convenient and keeps these rarely changing modules out of the way.
Test plan: It's a file move, `yarn test` suffices.
The logic for converting weights into an edge evaluator should not be
coupled to the frontend application.
Progress towards #967.
Test plan: Very straightforward rename; `yarn test` suffices.
Now that the `analysis` module owns the Node and Edge types, it should
own the "fallback plugin" too. (Note that it's not actually a plugin,
though it somewhat acts like one.)
We now declare the fallback type in `analysis`, along with a fallback
analysis adapter. `app/adapters` then declares a fallback app adapter.
Test plan: `yarn test`
Progress towards #967.
There's a folder called `app/credExplorer/weights` which contains the
type specification for weights (for PageRank configurability), and also
contains frontend code for specifying those weights. This commit creates
a `weights` module under `analysis` which will contain just the logic
for specifying and using the weights, without any frontend
consideration.
It's mostly a port of the existing logic in `credExplorer/weights`, with
the caveat that app adapter related concepts have been removed, in favor
of referencing the declaration instead.
We then remove the duplicated logic and re-route imports.
Test plan: `yarn test`
* Add the demo plugin
This ports the ad-hoc demo adapter defined in
`src/app/adapters/demoAdapters.js` into its own demo plugin.
This has the benefit that the demo plugin can now be depended on outside
the app module, e.g. for the analysis module as well. Correspondingly,
I've added a demo analysis adapter.
Test plan: `yarn test`. Note that no unit tests were added, as the demo
plugin is trivial.
* Delete `src/app/adapters/demoAdapters.js`
Now that we have an explicit demo plugin at `src/plugins/demo/`, we can
remove the legacy declaration of that plugin within the `app` module.
This commit deletes the old version, and re-writes all references to
point to the standalone plugin.
Test plan: `yarn test`
Summary:
It is time. (Replaced with #622.)
Test Plan:
Running `yarn flow` suffices. Running `yarn test --full` also passes.
wchargin-branch: remove-legacy-graphql
William Chargin
Dandelion Mané
Eric Ronne
Robin van Boven
Tyler Mace
Brian Litwin
Seth Benton
Ana Noemi
expravit
Ian Darrow
