Summary:
This commit approximately completes the implementation of #643.\* Plugin
adapters are now provided an `Assets` object at `load` time, which they
can use to resolve their plugin-specific API routes.
\* “Approximately” because there are some non-essential pieces of legacy
code that should be cleaned up.
Test Plan:
Unit tests modified, but it would be good to also manually test this.
Run `./scripts/build_static_site.sh` to build the site to, say,
`/tmp/gateway/`. Then spin up a static HTTP server serving `/tmp/` and
navigate to `/gateway/` in the browser. Note that the entire application
works.
wchargin-branch: use-assets-in-PluginAdapters
Summary:
This commit is the next step in #643. It makes the `RepositorySelect`
robust to being hosted at arbitrary gateways by accepting `Assets` and
resolving the repository registry API route appropriately.
Test Plan:
Unit tests modified, but it would be good to also manually test this.
Run `./scripts/build_static_site.sh` to build the site to, say,
`/tmp/gateway/`. Then spin up a static HTTP server serving `/tmp/` and
navigate to `/gateway/` in the browser. Note that you can navigate
around the application and load the repository registry on the prototype
without any console warnings or errors, although you cannot yet load
actual graph data.
wchargin-branch: use-assets-in-RepositorySelect
Summary:
This commit takes the next step toward #643 by exposing `Assets` to our
React components at top level. Components will be expected to pass them
down as appropriate; this commit does not add any actual uses.
Test Plan:
Apply the following patch:
```diff
diff --git a/src/app/Page.js b/src/app/Page.js
index 24c2602..7ac2641 100644
--- a/src/app/Page.js
+++ b/src/app/Page.js
@@ -24,6 +24,10 @@ export default class Page extends React.Component<{|
<Link to="/" className={css(style.navLink, style.navLinkTitle)}>
SourceCred
</Link>
+ <img
+ alt="fav"
+ src={this.props.assets.resolve("/favicon.png")}
+ />
</li>
{routeData.map(({navTitle, path}) =>
NullUtil.map(navTitle, (navTitle) => (
```
Then, observe that the favicon loads correctly and updates across page
loads and refreshes in the following situations:
- under `yarn start`;
- after building the static site and serving from root;
- after building the static site and serving from another gateway.
wchargin-branch: use-withAssets
Summary:
This is the last piece of major infrastructure for #643. It will enable
components like `Page` and `CredExplorerApp` to receive `Assets` as a
prop.
A previous iteration of the same functionality used the new Context API
in React v16.3. This did a good job of solving the problem in production
code, and was convenient. However, it is currently intractable to test
with the current state of Enzyme. It’s plausible that this might improve
in the future, so if threading down the props becomes to onerous, we
might check in to see how our testing libraries are doing. I expect that
the threading should not be too bad, given that we do the same thing
with `localStore`, which has worked (as far as I’m aware) without a
hitch.
Test Plan:
Unit tests added; `yarn test` suffices.
wchargin-branch: withAssets
Summary:
This is copied from the Sharness repository’s `test/.gitignore`, and is
also the same as Git’s `t/.gitignore` minus a Git-specific exclusion.
Suggested by @decentralion.
Test Plan:
Run `yarn sharness` and interrupt it (SIGINT) while `make` is running.
Note that `sharness/` now contains a `trash directory.*`, which Git
ignores.
wchargin-branch: sharness-gitignore
Summary:
As the next step for #643, this patch enables the app to be rendered at
non-root gateways by incorporating the relative-path history
implementation developed in #666. The app is not fully functional:
our React components do not yet know how to resolve assets, and so
fetches of resources like the repository will be against the wrong URLs.
Test Plan:
- Note that `yarn start` still works.
- Run `./scripts/build_static_site.sh` to build the site into, say,
`/tmp/gateway`.
- Run a static web server from `/tmp/gateway/` and note that (a) the
paths listed in the page source are relative, and (b) everything
works as intended, with no console messages in either Chrome or
Firefox.
- Run a static web server from `/tmp/` and navigate to `/gateway/` in
the browser. Note that the app loads properly, and that refreshes
work (i.e., the `pushState` paths are real paths). Note that the
repository registry cannot yet be loaded, and so PageRank cannot be
run.
wchargin-branch: relative-router
Summary:
This is the first observable step toward #643. Assets whose paths are
known as literals at server-side rendering time are now referenced via
relative paths. This means that the favicon and JavaScript bundle can be
loaded from an arbitrary gateway. The actual bundle code will still only
work when loaded from `/`.
This commit stands alone so that the enclosing change to the Webpack
config can be in as small a change as possible.
Test Plan:
- Note that `yarn start` still works.
- Run `./scripts/build_static_site.sh` to build the site into, say,
`/tmp/gateway`.
- Run a static web server from `/tmp/gateway/` and note that (a) the
paths listed in the page source are relative, and (b) everything
works as intended, with no console messages in either Chrome or
Firefox.
- Run a static web server from `/tmp/` and navigate to `/gateway/` in
the browser. Note that the favicon and JavaScript are correctly
noted, but that the router raises an error because it is trying to
load a non-existent route. (This behavior is unchanged.)
wchargin-branch: relative-lexically-static
Previously, expanding a node would display the individual connections
that contributed cred to that node. For nodes with high degree, this was
a pretty noisy UI.
Now, expanding a node displays "aggregations": for every type of
adjacent connection (where type is the union of the edge type and the
adjacent node type), we show a summary of the total cred from
connections of that type. The result is a much more managable summary
view. Naturally, these aggregations can be further expanded to see the
individual connections.
Closes#502.
Test plan: The new behavior is unit tested. You can also launch the cred
explorer and experience the UI directly. I have used the new UI a lot,
as well as demo'd it to people, and I like it quite a bit.
For the group of aggregations returned by aggregation operation (e.g.
the set of aggregations returned by a call to `flatAggregate`), the keys
are unique.
Test plan: `yarn test`
The TableRow currently has some margin on the left, but not on the
right. This is visually unbalanced, especially when expanded so depth>0,
as the content on the right is at the very edge of the shaded rectangle.
This commit cleans that up a bit!
Test plan: Visual inspection (see screenshots in the pull request). I
don't think unit tests are necessary for small visual tweaks like this.
Summary:
This is necessary for #643. If we’re serving `/prototype/index.html`, we
need to to use `..` to refer to the root of the site. This patch adds
`rootFromPath`, which performs the relevant transformation. (The
implementation is trivial, but figuring out exactly what the
specification should be was not!)
Test Plan:
Unit tests added; `yarn test` suffices.
wchargin-branch: rootFromPath
Summary:
This will enable clients to obtain the path to a static asset, even when
the app is not hosted at the root of a server, as outlined in #643.
This module will be used for simple assets (images, etc.) and API data
(fetches from `/api/**`) alike.
This supersedes #663. It includes the logic from that PR (`Assets`)
without the React-specific context bindings (`AssetsContext`).
Test Plan:
Unit tests included; `yarn test` suffices.
wchargin-branch: assets-resolver
Summary:
See #643 and the module docstring on `createRelativeHistory.js` for
context and explanation.
This patch adds `history@^3.0.0` as an explicit dependency—previously,
we were depending on it only implicitly through `react-router` (which
was fine then, but is not now). The dependency is chosen to match the
version specified in `react-router`’s `package.json`.
Test Plan:
Extensive unit tests included, with full coverage; `yarn test` suffices.
wchargin-branch: createRelativeHistory
This is preparation for #502 - we want to be able to describe groups of
nodes, e.g. "52 repositories", so we need a plural form for every node
name.
As a fly-by fix, I removed the parentheses around the node names in the
fallback adapter, as these proved to look ugly/inconsistent in the UI.
Test plan: `yarn test` is sufficient.
We humans tend to find information about humans more interesting than
information about commits or pulls. The UI should accomodate this by
defaulting to displaying GitHub user nodes in the cred explorer.
This is implemented as a new nullable argument to the PageRankTable. If
not present, then the filter defaults to showing all nodes. If the
default filter is present but doesn't match any available type, an error
is thrown.
Test plan: The new behavior is tested. Also, I checked it in the UI and
it works.
Closes#651
Previously, the ConnectionRow showed the score of the node that was the
source of the connection. I believe the UI will be more consistent and
useful if it instead shows the connection score, i.e. how important that
connection was to the node in scope. This combos well with PR #657.
Test plan: The change is very simple, and covered by unit tests. I also
verified the behavior by examining the cred explorer.
Summary:
This function normalizes paths like `foo/bar//../baz` to `foo/baz`. The
implementation is directly copied from Node’s source code, which is
available under an MIT License.
I looked for a suitable NPM package, and rejected `path-normalize` and
`normalize-path`. (The former is closer and explicitly purports to be
the right thing, but actually isn’t.)
Test Plan:
Unit tests added, with full coverage except for one branch; I include a
proof that the branch is unreachable.
Tested on Node v8.9.4, Node v10.0.0, and Node v10.8.0. Tests pass in
each case. In the latter two cases, inconsistencies between the
implementation and the actual `path.posix.normalize` would cause a test
failure. In the former case, they do not. (All such cases verified.)
wchargin-branch: path-normalize
Currently, the PagerankTable creates components in the following
pattern:
```
NodeRow (depth=0)
> ConnectionRowList (depth=1)
> ConnectionRow (depth=2)
> ConnectionRowList (depth=3)
```
This commit changes the cycle to the following:
```
NodeRow (depth=0)
> ConnectionRowList (depth=0)
> ConnectionRow (depth=0)
NodeRow (padding=true, depth=1)
> ConnectionRowList (depth=1)
> ConnectionRow (depth=1)
```
This has some nice properties:
First, the context visually resets every time we return to a NodeRow, which
makes it feasible to change the score column to always have a context
dependent meaning:
- for a node row, the score is the score of that node.
- for a connection row, the score is the score contribution of that
connection
- (as of #502): for an aggregation row, the score is the score
contribution of that aggregation
We think this will be visually clear thanks to the padding around the
new NodeRow, along with the new color block indicating a new scope.
This design also ensures that every NodeRow has the full width available
to it (rather than getting crushed into a progressively smaller area of
the table), which will be very convenient for when we add renderers for
the nodes.
Thanks to @theliamcrawford as the idea for this came during a user study
with him.
Test plan:
The updated unit tests should be comprehensive. Also, try expanding some
rows in the cred explorer and verify that the behavior is as described.
William and I were experimenting, and felt that this color is slightly
more pleasing / harmonious with the rest of the site, and still quite
legibile.
Test plan: Examine the new UI, and conclude that the color choice is
harmonious and legible :). Screenshot included with the PR.
Paired with @wchargin
Summary:
Due to <https://github.com/facebook/flow/issues/6400>, patches like the
following weren’t raising Flow errors:
```diff
diff --git a/src/app/adapters/adapterSet.test.js b/src/app/adapters/adapterSet.test.js
index 67dd3ed..ccc6ac6 100644
--- a/src/app/adapters/adapterSet.test.js
+++ b/src/app/adapters/adapterSet.test.js
@@ -77,6 +77,7 @@ describe("app/adapters/adapterSet", () => {
const x = new TestStaticPluginAdapter();
const fallback = new FallbackStaticAdapter();
const sas = new StaticAdapterSet([x]);
+ sas.wat();
return {x, fallback, sas};
}
it("errors if two plugins have the same name", () => {
```
A `flow type-at-pos` check indicated that the type of `sas` was indeed
inferred as `any`.
This patch applies the usual nonsensical fix. Better safe than spooky.
Test Plan:
The above patch now raises a Flow error.
wchargin-branch: annotate-adapterset-constructors
Summary:
See justification in the added unit test.
Test Plan:
Added unit test, with justification.
Also, `yarn sharness-full` passes, and `yarn start` still works.
wchargin-branch: route-trailing-slashes
This commit adds the `showPadding` prop to `TableRow`s. If showPadding
is true, then the row will have vertical padding above the row, and
below the last child of the row. The padding will match the background
color of the given row. The padding is implemented as extra `tr`
elements that themselves contain empty tds.
Test plan: The new behavior is pretty thoroughly covered by new unit
tests. Additionally, if you want to see padding in the live UI, you can
apply the following (slightly contrived) diff.
```
diff --git a/src/app/credExplorer/pagerankTable/Connection.js b/src/app/credExplorer/pagerankTable/Connection.js
index 3a882cd..633525b 100644
--- a/src/app/credExplorer/pagerankTable/Connection.js
+++ b/src/app/credExplorer/pagerankTable/Connection.js
@@ -70,7 +70,7 @@ export class ConnectionRow extends React.PureComponent<ConnectionRowProps> {
depth={depth}
description={connectionView}
connectionProportion={connectionProportion}
- showPadding={false}
+ showPadding={depth % 3 === 0}
score={sourceScore}
>
<ConnectionRowList
```
Currently, as we expand nodes or connections in the PagerankTable, the
rows both get more indented and attain a deeper color. Both of these
behaviors are controlled by the `depth` parameter.
We're going to switch the UI to a cyclic structure, where as you drill
down, once you get back to a particular node, the indentation resets to
base, but the color - which now indicates nested depth - continues to
change. This commit sets that change up, by splitting the behvaior into
two parameters: `depth`, which controls the color, and `indent`, which
controls the indentation level.
As a small additional tweak, the indentation formula is changed so that
buttons are always indented by 5 pixels. This results in a cleaner
display for nodes that have `depth>0` but `indent==0` (as the button
doesn't look squahsed against the color boundary).
Test plan:
The change is very simple; inspecting the updated snapshots should be
persuasive.
We currently have two components which create rows in our PagerankTable:
the `NodeRow` and `ConnectionRow`. Work on #502 will result in the
addition of a new one, the `AggregationRow`. It's time to stop
duplicating logic (and testing) of the shared behavior for these rows,
like depth-based styling, row expansion/collapse, etc. This commit pulls
all the common logic to rendering rows into a single, thoroughly tested
`TableRow` component.
There is one observable change in the UI: when a connection percentage
is not available (i.e. for NodeRows), we now leave the column empty
rather than placing a dash in the column. I think this is visually
cleaner.
Test plan: Unit tests pass, and this part of the code base is thoroughly
tested, so that's a pretty reliable indicator. I also poked around the
live PagerankTable in the cred explorer, just to be safe.
When using Tries, we often want the last matching entry for the given
path, and to throw an error if one is not available. By adding this
method to the API, we avoid a lot of unnecessary repetition in the code
base.
Test plan: Unit tests pass. As this touches the untested WeightConfig,
I've also manually tested the weight config behavior.
Summary:
An `import *` was used for convenience, but this effects a value import
in addition to a type import. By exploding the wildcard import to
directly import the required types, we can shave off 2.3% of our
post-gzip bundle size (131.82 KB to 128.74 KB). It’s unfortunate that we
lose the namespacing, but c’est la vie.
Test Plan:
`yarn flow` suffices.
wchargin-branch: explode-wildcard-type-import
Thanks to #642, it should now be safe to disable the Git plugin, reaping
the benefits described in #628, without causing the cred explorer to
crash (#631).
Test plan:
- `yarn travis --full` passes
- The full cred explorer works:
- Running PageRank does not crash the explorer
- Expanding a pull request does not crash the explorer
- (After clearing state) the weight config doesn't show Git weights
- The filter doesn't show Git nodes
This modifies WeightConfig to properly use the fallback node type, as
created in #640 and merged in #642. As an additional change, it now
displays type names, rather than the address parts. For example, the
issue type is now displayed as `Issue`, not
`["sourcecred", "github", "issue"]`.
The WeightConfig is an untested mess, and I will likely re-write it
entirely. (See a bevy of WeightConfig related issues: #604, #595, #588).
So, not too much effort was invested in keeping high code quality in
this commit.
Test plan: The weight config has no tests, so I manually tested:
- weights persist after page reload
- node weights influence cred attribution
- edge weights influence cred attribution
- edge directionality influences cred attribution
- weights have reasonably pretty description messages
This takes the code from #640 and puts it into production.
Test plan: Unit tests pass. The observable behavior in the cred explorer
is unchanged; i.e. the addition of the FallbackAdapter did not produce
new entries in the WeightConfig or in the Pagerank table options. The
WeightConfig is untested, so we don't have verification of that behavior
(other than that I tested it and am reporting it here). The
PagerankTable code is tested, and a snapshot would fail if another
option group had appeared.
Issue #631 revealed that our current plugin-handling code is fragile -
we aren't robust to having nodes from a plugin without having that
plugin present in the frontend. This commit adds `StaticAdapterSet` and
`DynamicAdapterSet`, which are abstractions over finding the matching
plugin adapter or type for a node or edge. It's a robust abstraction,
because the adapter sets always include the `StaticFallbackAdapter` or
`DynamicFallbackAdapter`, which can match any node, so we'll never get
an error like #631 due to not having an adapter / type available.
Also relevant: #465
Test plan:
Unit tests included.
Summary:
This patch considers an environment variable `GITHUB_DELAY_MS`. If the
value is set to a positive integer, a delay of the given number of
milliseconds will be incurred before each query to GitHub. This is to
decrease the probability of being rate-limited; see #350 for details.
This in turn unblocks us to load SourceCred data for larger
repositories.
The use of an environment variable is something of a hack to get this
off the ground. See #638 for long-term plans.
Test Plan:
Run `time node ./bin/sourcecred.js load sourcecred/example-github` with
varying values for `GITHUB_DELAY_MS`. Note that with the variable unset,
set to zero, set to a negative number, or set to a non-numeric value,
the job completes quickly; when the delay is set to `5000`, the job
takes an extra five seconds.
wchargin-branch: delay-github-queries
For #502: The UI that I currently have in mind displays aggregations
grouped by connection type and node type together, rather than nested.
I think it will be cumbersome to have multiple hierarchical
levels of expansion.
To make that UI easy to write, this commit adds some logic for
flattening the hiearchical aggregation from #624. I add an extra
translation to flatten, rather than just having the logic produce nested
structures, because it's convenient to keep around the nested structure
in case I decide to implement the hierarchical UI instead. Once we have
solidified how we want the UI to behave, we might choose to simplify
this code.
Test plan: The implementation is rather simple. There are some unit
tests.
This commit creates the directory `src/app/adapters` and moves the
following three files into it:
- `src/app/pluginAdapter.js`
- `src/app/pluginAdapter.test.js`
- `src/app/defaultPlugins.js`
This is in preparation for a principled fix for #631, which will add a
base plugin and some logic that ensures it's always included.
Summary:
In addition to the obvious benefit of having a favicon, this gets rid of
a 404 Not Found error on our home page, tremendously boosting our hacker
cred.
Test Plan:
The favicon is displayed in both `yarn start` and the static site (as a
result of the build script). The added build test fails before this
change.
wchargin-branch: add-favicon
Summary:
The SVG was artisanally crafted by yours truly, and rasterized by the
accompanying script (which is fully deterministic, and also artisanal).
Test Plan:
Run `./src/assets/logo/rasterize.sh`, and note that the output is
unchanged.
wchargin-branch: add-logo
Summary:
This is a follow-up to #514, wherein we disabled new service workers and
instructed any existing service workers to self-destruct. (See that PR
for the rationale.) This commit removes them from our codebase entirely,
enabling us to slim down our build process and our build output.
Test Plan:
Running `yarn start` still works. Building the static site and exploring
it works, too.
wchargin-branch: remove-sw
Summary:
This fixes the following warning on our development instance:
> Warning: render(): Calling ReactDOM.render() to hydrate
> server-rendered markup will stop working in React v17. Replace the
> ReactDOM.render() call with ReactDOM.hydrate() if you want React to
> attach to the server HTML.
We do in fact want to attach to the server HTML, so we apply the
suggested patch.
(The warning of course also applies to production, but warnings do not
appear in production.)
Test Plan:
Running `yarn start` shows that the above warning has disappeared, and
that the cred explorer still works. (Also, `yarn test --full` passes,
but that tells us effectively nothing because this code path is never
hit in tests: it only affects the HTML that is executed in the browser.
Erasing the entire module, leaving only `// @flow`, still lets tests
pass.)
wchargin-branch: migrate-to-hydrate
This reverts commit 8c70f03122.
Context: This introduced a serious bug (#631), so we're reverting it to
get the codebase back in a working state. Meanwhile, I'll develop a
principled solution.
Test plan:
I rebuilt the backend, re-loaded a graph, and loaded it in the frontend.
PageRank, the cred explorer, and the weight config all work. Opening a
pull request does not trigger a crash.
This implements two methods:
`aggregateByNodeType` groups `scoredConnection`s by the specified
`NodeTypes`, along with summary statistics.
`aggregateByConnectionType` groups `scoredConnection`s by
`ConnectionType` at the top level, where `ConnectionType` includes
`EdgeType` and direction, (and also captures synthetic self-loops).
Then it also groups by `NodeType` within any aggregation.
This is progress towards #502.
Test plan: unit tests included.
See #627 for context.
Removing the Git plugin results in an enormous performance improvement.
In my testing on `metamask/metamask-extension`: before this change, load
took 23s, and PageRank took >9 mins and then crashed. After this change,
load+PageRank took 5s combined.
Test plan: All unit tests pass; loading new data from the CLI works; and
I poked around the UI to make sure there were no spurious references to
the Git plugin.
Note: This does not break backcompat, there's no need to regenerate any
already-loaded data.
This is the first real step towards #502.
Factoring this out because deciding the type signatures was non-trivial,
and the work was paired with @wchargin.
Test plan: `yarn test`
PagerankTable is getting a bit unwieldy, especially as #502 will need to
add a new pair of components. This commit splits the erstwise
PagerankTable.js into four files:
- `pagerankTable/shared`, shared utils and types
- `pagerankTable/Node`, the `NodeRow` and `NodeRowList`
- `pagerankTable/Connection`, the `ConnectionRow`, `ConnectionRowList`,
and `ConnectionView`
- `pagerankTable/Table`, the `PagerankTable` itself
This commit makes no logical changes; it is purely a reorganization.
Test plan: `yarn test`
PageRank outputs scores as components in a probability distribution.
This means that most scores are very small numbers, e.g. 0.00003. This
doesn't make for a great UI (humans don't like thinking in tiny
decimals).
Our first attempt to come up with a more readable UI was to use log
scores; in #265 we displayed the log score alongside (arbitrarily)
`rawScore * 100` in the UI. The log scores were more usable, so we kept
them, with subsequent modifications. In the original version, all the
log scores were negative. In #466, we arbitrarily added 10 to the
scores, which made most scores look nicer, but introduced a meaningless
switch where scores counter-intuitively become negative after a certain
point. That was bad, so in #535 we started displaying negative log
scores. This is also counter-intuitive: it's weird that lower scores are
better, and it's not clear that a score of (say) 3 is 20x better than a
score of 6.
I think we need to do away with the log scores; people just don't think
about numbers logarithmically. This commit switches to linear scores,
normalized so that the largest score is always 1000. I've tried this out
on a few repos and demo'd it to people, and it seems much clearer.
Test plan: Some unit tests added; also, I launched the cred explorer and
experienced the change on several projects.
We manually import the set of plugins to be used by the app in a few
different places. That's a bad smell. This commit creates a centralized
import point instead.
Test plan: `yarn test`
This commit adds some consistent and tested methods for getting the
appropriate plugin adapter for a given Node/Edge address. There are
methods for both static and dynamic adapters.
In the event that more than one plugin adapter matches the given
address, an error is thrown; likewise in the case where there is no
matching adapter.
Test plan: `yarn test`
Relevant to #465
Summary:
The `node ./bin/sourcecred.js load` command invokes plugin code by
providing an output directory into which the plugin may store data.
As of this patch, it also provides a cache directory that the plugin may
use to store data that will not be available at runtime. For instance,
the Git plugin might choose to clone the repository herein, or the
GitHub plugin may choose to store partial GraphQL query results to deal
with interruptions. The contract is that the cache directory may be
removed at any time and that the plugin should continue to operate
normally.
Test Plan:
The build script has been updated and tested. Reverting the change to
the build script causes the newly added test to fail. (Each plugin has a
cache directory, though the cache directories are empty for now.)
wchargin-branch: create-plugin-cache
It's annoying that it always appends it to the end of the PR message
when we've uploaded a single commit. Since we have new contributors
relatively rarely, but we are slightly annoyed by it constantly, it's
better to remove it.
Test plan: not needed
William Chargin
Dandelion Mané