1416 Commits

Author SHA1 Message Date
Dandelion Mané
cb1d612f1f
Add the CredResult module (#1830)
This module has a concise and clean type for storing all the output data
from running SourceCred, including:
- The graph
- The weights
- The cred scores and flows
- The interval timing info
- The plugins used
- The parameters used

We also have a method `compute` which computes one of these given the
weighted graph, parameters, and plugins. It's all quite clean and
simple. I feel good about thsi API.

Test plan:
The main `compute` function is only sanity checked via flow and unit
testing, which is appropriate since it's a pure-piping function with
little than can go wrong. I've also added JSON serialization; this is
tested with round trip testing.
2020-06-01 17:47:20 -07:00
Dandelion Mané
c5c9f950d4
Compute CredData from TimelineCredScores (#1831)
This commit adds the `analysis/credData` module, which processes raw
TimelineCredScores into a format which is better for serialization and
data analysis. In particular, this format explicitly stores the summary
(summed-across-time) data for nodes separately from the raw temporal
data, which will allow us to throw away raw data for uninteresting or
low-cred nodes, while keeping the summary.

Test plan: I've added some basic unit tests; run `yarn test`.
2020-06-01 17:14:26 -07:00
William Chargin
ae181c2fda
combo: add exactly combinator for fixed values (#1829)
Summary:
Write `C.exactly(["red", "green", "blue"])` to admit any of a fixed set
of primitive values. This can already be implemented in terms of `raw`
and `fmap`, but it’s more convenient to have a dedicated helper.

Test Plan:
Unit tests included, retaining full coverage.

wchargin-branch: combo-exactly
2020-05-31 22:33:27 -07:00
William Chargin
ee0beaf1f1
combo: add backtracking orElse combinator (#1828)
Summary:
The `orElse` combinator tries parsing the input with a each of a
sequence of parsers, taking the first successful match (if any). It’s
handy for parsing union types; see examples in the patch.

This combinator treats all errors as recoverable. For instance, a parser

```javascript
C.orElse([
  C.object({type: "NUMBER", number: C.number}),
  C.object({type: "STRING", string: C.string}),
  C.object({type: "BOOLEAN", boolean: C.boolean}),
]);
```

on input `{type: "NUMBER", number: "wat"}` will see that the first
parser fails and try all the others, even though we can see that as soon
as any of these parsers matches the `type` field, we can commit to that
parser rather than backtracking on failure. In a pathological case, even
a small parser can run very slowly and generate very long errors:

```javascript
    it("big", () => {
      let p = C.null_;
      for (let i = 0; i < 12; i++) {
        p = C.orElse([p, p]); // exponential in loop iteration count
      }
      const result = p.parse("non-null!");
      if (result.ok) throw new Error("unreachable");
      expect(result.err.length).toEqual(22558015);
    });
```

Hopefully, this won’t be a problem in practice.

Test Plan:
Unit tests included, retaining full coverage.

wchargin-branch: combo-orelse
2020-05-31 22:29:38 -07:00
William Chargin
0dae2346fc
combo: add raw identity parser (#1827)
Summary:
If you want to parse an arbitrary JSON object, or if you want to parse a
complicated dynamic type that can’t easily be expressed in terms of the
other base parsers and combinators, use `C.raw`. This combinator accepts
any input and returns it as a `JsonObject`—not an `any`, so it’s still
safely typed, but the client (either an `fmap` transform or the consumer
of the eventual `parse`/`parseOrThrow`) will need to destructure it by
pattern matching on the `JsonObject` structure.

Test Plan:
Unit tests included (though full coverage technically doesn’t require
them).

wchargin-branch: combo-raw
2020-05-31 22:24:17 -07:00
William Chargin
500140d292
combo: add dict combinator (#1826)
Summary:
This is for homogeneous object types with unbounded key sets: roughly,
`dict` is to `object` as `array` is to `tuple`. Its implementation
requires no terrifying type magic whatsoever.

Test Plan:
Unit tests included, retaining full coverage.

wchargin-branch: combo-dict
2020-05-31 22:21:10 -07:00
William Chargin
205f6e064c
combo: add tuple combinator (#1825)
Summary:
This new combinator parses heterogeneous tuples: arrays of fixed length,
which may have different, fixed types at each index. For example:

```javascript
type CompactDistribution = [NodeAddressT, number][];
function compactDistributionParser(): C.Parser<CompactDistribution> {
  return C.array(
    C.tuple([
      C.fmap(C.array(C.string), NodeAddress.fromParts),
      C.number,
    ])
  );
}
```

Or:

```javascript
function compatParser<T>(underlying: C.Parser<T>): C.Parser<Compatible<T>> {
  return C.tuple([
    C.object({type: C.string, version: C.string}),
    underlying,
  ]);
}
```

Test Plan:
Unit tests included, retaining full coverage.

wchargin-branch: combo-tuple
2020-05-31 22:17:54 -07:00
William Chargin
64169da128
discourse: use parser combinators for config (#1824)
Summary:
This patch changes the Discourse plugin’s config parsing to use the
parser combinator approach, which simplifies it and removes the need for
all the tests.

Test Plan:
Prior to deleting the tests, they passed, except for the test that
mirror options could not be partially populated; the partial parsing now
works correctly.

wchargin-branch: discourse-config-combinator
2020-05-31 21:52:05 -07:00
William Chargin
930ac715fa
combo: add shape helper utility (#1823)
Summary:
Users may now write `C.shape(fields)` instead of `C.object({}, fields)`.

Test Plan:
Unit tests included.

wchargin-branch: combo-shape
2020-05-31 21:48:23 -07:00
Dandelion Mané
4635b46d1f
distributionToCred propagates cred flow data (#1822)
This commit modifies the distributionToCred module so that in addition
to normalizing the scores into cred, it also normalizes the score flows
(from #1802) into cred flows. This is another big part of #1773, so that
we can support the `OutputEdge` format and give meaningful info on cred
flows for each edge.

The change is quite simple, as we already compute a normalization
constant, we just need to apply it to a few more arrays.

Test plan: I added a new unit test that has values in all of the
different flows, so we can validate that they are normalized
consistently.
2020-05-30 16:44:07 -07:00
William Chargin
109ebb0417
combo: convert instance and GitHub configs (#1820)
Summary:
This patch converts the `instanceConfig` and `github/config` parsers
from hand-written traversals to combinator-based structures. The
resulting code is much simpler and easier to read.

Test Plan:
Running the `sc2` CLI for `load` and `graph` still works in the happy
path, with the expected errors if keys in the config files are changed
to the wrong strings.

wchargin-branch: combo-instance-github
2020-05-30 16:04:04 -07:00
William Chargin
272af9db38
combo: allow field renaming (#1819)
Summary:
This patch expands the API of `Combo.object` such that fields in the
input JSON may be renamed in the output JSON. This often occurs in
combination with `fmap`, as we convert a simple string field with a
user-facing name to a different structured representation. For example:

```javascript
C.object({
  repoIds: C.rename(
    "repositories",
    C.fmap(C.array(C.string), repoIdToString)
  ),
});
```

This is backward-compatible and invisible when not needed: the fields of
the argument to `C.object` may now be either parsers (as before) or
results of `C.rename`.

This patch also adds a check that the required and optional key sets
don’t overlap, which could technically have happened before but is more
important now that renames are possible.

Test Plan:
Unit tests included, retaining full coverage.

wchargin-branch: combo-rename-fields
2020-05-30 15:58:05 -07:00
William Chargin
ce578ef84c
combo: support heterogeneous objects (structs) (#1818)
Summary:
This combinator is critical to parsing real-world types, which almost
always contain objects at some level. This combinator supports objects
with both required keys and optional keys, as is often needed for
parsing config files and the like.

There’s a bit of dark Flow magic here. In particular, I’ve found that
using a phantom datum in the parser type is necessary for Flow to
correctly extract the output type of a parser without collapsing
discriminated unions. I also ran into some bugs with `$Shape`, which are
avoided by using `$Rest<_, {}>` instead, per this helpful user comment:
<https://github.com/facebook/flow/issues/7566#issuecomment-526324094>
Finally, Flow’s greedy inference for multiple-arity functions causes it
to infer and propagate an empty type in some cases, so we type `object`
as an intersection of functions, or, equivalently, an interface with
multiple callable signatures; this is a trick borrowed from the standard
library for functions like `filter`.

Critically, the tests for this module include tests for expected Flow
errors, so we should be reasonably safe against regressions here.

Test Plan:
Unit tests included, retaining full coverage.

wchargin-branch: combo-object
2020-05-30 15:55:00 -07:00
William Chargin
c5bdfcd174
combo: add pure and fmap combinators (#1817)
Summary:
These are commonly used for building large parsers from small pieces,
especially `fmap`, which is instrumental for transformation and
validation. We pick the name `fmap` rather than `map` to avoid confusion
with ES6 `Map` values, which are unrelated.

Test Plan:
Unit tests and type-checking tests included, with full coverage.

wchargin-branch: combo-pure-fmap
2020-05-30 15:51:08 -07:00
William Chargin
297c4e9156
combo: begin simple parser combinator library (#1816)
Summary:
We often want to parse data from JSON files on disk into similar object
structures in memory. But `JSON.parse` is untyped both statically and
dynamically: it has type `(string) => any`, and it’s happy to accept
structures that aren’t in the shape that you expected. Whenever we write
something like `const c: MyConfig = JSON.parse(raw)` where `raw` comes
from a user-editable file on disk, we’re introducing a trivial soundness
hole. Furthermore, we often want to use a different in-memory state from
the serialized form: perhaps we use ES6 `Map`s in memory, or perhaps
we’ve refined a raw string type to an opaque validated type like
`RepoId` or `NodeAddressT`. These can be done by manually walking the
output of `JSON.parse`, but it’s not pretty: see `instanceConfig.js` or
`github/config.js`.

Parser combinators are a solution to this problem that enable building
parsers for simple primitives and composing them to form parsers for
larger structures. This patch introduces the skeleton of a parser
combinator library, supporting JSON primitives and arrays (but not
objects) along with tests that show its usage. Support for heterogeneous
object (“struct”) types will come in a subsequent patch because the
typing implementation is more complicated, though the interface to
clients is just as simple.

For comparison, this is essentially the `FromJSON` half of the Haskell
library [Aeson][aeson].

It’s possible that we’ll want to generalize this to a broader system of
profunctor optics, maybe over monad transformers, which would make it
easier to both parse and serialize these structures (using “isos” rather
than just parsers everywhere). But manually serializing the structures
is easier than manually parsing them, because they start out strongly
typed. The profunctor generalization is more complicated, and in the
meantime this solves a useful problem, so let’s defer the generality
until we decide that we need it.

[aeson]: https://hackage.haskell.org/package/aeson

Test Plan:
Unit tests included, with full coverage.

wchargin-branch: combo-init
2020-05-30 15:43:11 -07:00
Dandelion Mané
40426f353c
Implement bare-bones sc2 merge and score (#1815)
This adds the missing `merge` and `score` commands to the sc2 CLI.
`merge` currently just merges the plugin graphs, and doesn't yet support
identity resolution or weight overrides.

`score` computes the new data output format (cf #1773) and writes it to
disk. It doesn't yet support using custom parameters.

Test plan:
Follow the test plans for the previous commits, then run `sc2 merge`.
It will create a combined graph at `output/graph.json`, which will
contain data for both Discourse and GitHub. Then run `sc2 score` and
the `output/cred.json` file will contain scores for the combined
graph.
2020-05-30 15:26:01 -07:00
William Chargin
c2510d0c1d
cli2: exit on invalid subcommand invocation (#1821)
Summary:
We’d written `die("usage: ...")`, but forgotten to actually return.

Test Plan:
After patching a test instance’s `sourcecred.json` to be invalid,
running `sc2 load extra-arg` now only prints a usage message rather than
an invalid configuration message.

wchargin-branch: cli2-actually-die
2020-05-30 15:20:13 -07:00
Dandelion Mané
7a9d04bc60
cli2: add Discourse CLI plugin (#1814)
This updates the v2 CLI so that it now supports the Discourse plugin.

Test plan:

Modify the test instance described in the previous commits so that the
root `sourcecred.json` file includes "sourcecred/discourse" in the list
of bundled plugins. Then, add a
`config/sourcecred/discourse/config.json` file with the following
contents:
`{"serverUrl": "https://sourcecred-test.discourse.group/"}`

Now, running `sc2 load` will load Discourse data, and `sc2 graph` writes
a Discourse graph in the output directory.
2020-05-30 14:43:50 -07:00
Dandelion Mané
5279c34c6a
Expose edge-level score flows in timelinePagerank (#1802)
This commit makes a big step towards realizing the v3 output format
(see #1773). Specifically, it modifies the timelinePagerank interval
result format so that in addition to the distribution (the score for
each ndoe), we also track:
- How much score flowed to each node from the seed
- How much score flowed across each edge in a forwards (src->dst)
direction
- How much score flowed across each edge in a backwards (dst->src)
direction
- How much score flowed to each node from its synthetic self loop

The result is that we can now precisely decompose where a node's score
came from (and where it went to). Specficially, for every node we have
the invariant that the node's score is equal to the sum of its seed
score, its synthetic loop score, and the forward flow on each edge for
which the node was dst, and the backward flow for each edge on which the
node was src.

Test plan:
I've added unit tests which verify that the invariant above holds for
real PageRank results on a small example graph.
2020-05-30 14:37:23 -07:00
Dandelion Mané
27aeacc906
Factor out _intervalResult in timelinePagerank (#1799)
As part of work for #1773, I want to add a lot more complexity to the
logic for computing individual time-slices of pagerank scores, so that
we can trace how much score flowed on each individual edge. This means
adding more complexity to the _computeTimelineDistribution function;
however, that function is an untested wrapper, so I'm hesitant to add
more complexity directly.

Instead, I'm first factoring out an _intervalResult method and a
corresponding type, which computes the scores for a given timeline
interval. I've also added sanity checking tests for this method. In a
followon commit, I'll add more logic for tracking edge-level score
flows.

Test plan: This is just a refactor, maintaining existing behavior and
adding tests. `yarn test --full` passes. Since our sharness tests
include doing a full load (including timeline cred computation) on
realistic data from GitHub, this gives us confidence that there hasn't
been any change to cred semantics.
2020-05-30 14:14:32 -07:00
Dandelion Mané
71c0b0d66d
Refactor timelinePagerank, exposing connections (#1797)
This commit refactors internal helper methods in timelinePagerank so
that rather than piping around an OrderedSparseMarkovChain, we instead
provide the NodeToConnections from which that OSMC may be derived. This
is important because the NodeToConnections has the information necessary
to derive how score flowed across individual edges, and not just on the
adjacency topology of the graph. This will allow us to compute the
OutputEdge format with edge-specific cred flows as documented in #1773.

Test plan: `yarn test` passes. It's a simple refactor.
2020-05-30 14:02:23 -07:00
Dandelion Mané
d1144217b0
cli2: Use TaskReporters (#1813)
This commit modifies the cli2 interfaces so that plugins may use task
reporters when loading, generating a reference detector, or creating a
graph. Also, the scaffold now automatically reports on task/plugin-level
progress as appropriate.

Test plan: Generate an example instance as described in previous
commits, then run `load` and `graph` and get timing info:

```
~/tmp/instance❯ node $sc/bin/sc2.js load
  GO   load
  GO   loading sourcecred/github
  GO   github: loading sourcecred/example-github
 DONE  github: loading sourcecred/example-github: 220ms
 DONE  loading sourcecred/github: 221ms
 DONE  load: 227ms
~/tmp/instance❯ node $sc/bin/sc2.js graph
  GO   graph
  GO   reference detector
  GO   reference detector for sourcecred/github
 DONE  reference detector for sourcecred/github: 296ms
 DONE  reference detector: 297ms
  GO   sourcecred/github: generating graph
 DONE  sourcecred/github: generating graph: 242ms
 DONE  graph: 544ms
 ```
2020-05-30 12:10:46 -07:00
William Chargin
8d88ea937b
sc2: add graph subcommand (#1811)
Summary:
Paired with @decentralion.

Test Plan:
Follow the test plan for #1810, then additionally run

```
(cd /tmp/test-instance && node "$OLDPWD/bin/sc2.js" graph)
```

and note that the `output/graphs/...` directory has a graph JSON file.

wchargin-branch: cli2-graph
2020-05-28 19:01:03 -07:00
William Chargin
80c3c38282
cli2: add sketch of plugin loading (#1810)
Summary:
This adds a `CliPlugin` interface and a basic implementation for the
GitHub plugin.

Paired with @decentralion.

Test Plan:
Create a new directory `/tmp/test-instance`, with:

```
// sourcecred.json
{"bundledPlugins": ["sourcecred/github"]}

// config/sourcecred/github/config.json
{"repositories": ["sourcecred/example-github"]}
```

Then, run

```
yarn backend &&
(cd /tmp/test-instance && node "$OLDPWD/bin/sc2.js" load)
```

and observe that the new instance has a cache directory containing a
GitHub database.

wchargin-branch: cli2-load
2020-05-28 18:57:27 -07:00
Dandelion Mané
0f6a765569
Refactor TimelineCredScores data type (#1804)
This commit refactors the TimelineCredScores data type so it is an
array-of-objects rather than an object-of-arrays. I want to add several
more fields (for forward cred flow, backwards cred flow, seed flow,
synthetic loop flow), and feel it will be a lot cleaner with an
array-of-objects.

This is a refactor of a local data type, and there's test coverage.
Likelihood of regression is very low.

Test plan: Updated tests; `yarn test` passes.
2020-05-28 18:53:46 -07:00
William Chargin
0449c9ea37
cli2: add skeleton of new CLI (#1809)
Summary:
This patch creates a new binary, `./bin/sc2`, which will be the home for
a rewrite of the CLI intended to implement an instance system. See:
<https://discourse.sourcecred.io/t/sourcecred-instance-system/244>

Paired with @decentralion.

Test Plan:
Run `yarn backend && node ./bin/sc2.js`, which should nicely fail with a
“not yet implemented” message.

wchargin-branch: cli2-skeleton
2020-05-28 18:33:04 -07:00
Dandelion Mané
97da2ae077
Update initiatives plugin to warn on broken urls (#1806)
Currently, if the initiatives plugin is unable to resolve a url, it
simply silently ignores it. This can lead to very big errors in cred
with no clear cause.

We need to address the root cause, but for now this change just makes
the problem more visible--we now print a warning when this happens.

Test plan: When run on the real `MetaFam/TheSource` cred instance, we
now see a lot of warnins printed to console:

```
entry Gitcoin Donation (proofoftom): no address for contributor https://github.com/proofoftom
Gitcoin Donation (adamstallard): no address for contributor https://github.com/adamstallard
Gitcoin Donation (mccallj57): no address for contributor https://github.com/mccallj57
Gitcoin Donation (pet3r-pan): no address for contributor https://github.com/pet3r-pan
Gitcoin Donation (nidea4u): no address for contributor https://github.com/nidea4u
Tavern Keeping (community/DAO management): no address for https://github.com/MetaFam/TheSource/tree/master/initiatives/2020-02-setup-dao.json
Taking Notes: no address for contributor https://github.com/teochiriac
Initiating SourceCred Partnership: no address for https://forum.metagame.wtf/u/hammadj
DAO Setup: no address for https://github.com/pythonpete32
Character Art: no address for https://github.com/kerstentw
Beanow Discord Plugin: no address for contributor https://github.com/Beanow
Intercon: Moderating: no address for https://forum.metagame.wtf/u/FelipeDuarte
Intercon: Moderating: no address for https://github.com/DeFiDude
Intercon: Making NFT Badges: no address for https://github.com/DeFiDude
Intercon: Coordinating Speakers: no address for https://github.com/DeFiDude
Hubs Setup: no address for contributor https://github.com/drewharding
SourceCred Tech Support: no address for https://github.com/Beanow
Setting Up Initiatives/Quest Plugin: no address for https://github.com/Beanow``
```
2020-05-28 18:20:42 -07:00
Dandelion Mané
70a4597610
Update example-github snapshot (#1800)
Without this change, `yarn test --full` fails.
2020-05-18 12:08:57 -07:00
Dandelion Mané
91ca897d99
Update output format to include credOverTime (#1791)
As requested by @s-ben, we map now include cred over time for all
contributions, not just contributors. Based on discussion with @Beanow,
we made it an optional field so that we can optionally filter to save
space instead.

I was initially concerned that we wouldn't be able to compute
credOverTime for non-user nodes in CredRank, which is why I left it out.
However, informed by discussions with @mZargham, I'm less concerned
because PageRank (and thus CredRank) is a linear operator on the seed
vector. So, if we want to compute the "cred over time" for individual
contributions in CredRank, we can do so by constructing time-specific
seed vectors (which flow only to activity minting cred in the specified
interval), and the sum of contributions time-scoped cred will be equal
to the non-time-scoped cred. It's good that we'll still have the epoch
nodes for users, as that will allow us to model sponsorship cred flow
dynamics.

cc @wchargin for CredRank considerations.

Test plan: Unit tests updated, `yarn test` passes.
2020-05-15 11:12:15 -07:00
Dandelion Mané
ec56ce79c8
Use the less restrictive Timestamp type (#1790)
This commit builds on the change in #1789 which made Timestamp
validation optional. Now, it is convenient to use Timestamps
consistently across the codebase. This is split out from #1788.

Test plan: `yarn test` and `git grep "timestampMs: number"`. The only
hit is within the timestamp module itself.
2020-05-13 11:49:26 -07:00
Dandelion Mané
34bb471d38
Make TimestampMs validation optional (#1789)
Currently, the Timestamp module requires that all TimestampMs types be
validated via the `fromNumber` method. However, as discussed in #1788,
this creates marginal benefit (just knowing that a TimestampMs is an
integer does not give great confidence that it's a meaningful and
correct timestamp for any use case), and creates a lot of overhead,
especially in test code. It makes adopting the TimestampMs type across
the codebase an arduous and mostly thankless task that would involve
re-writing dozens or hundreds of test instances to add zero actual
safety.

Therefore, I've made the type non-opaque, and renamed `fromNumber` to
`validate`. I then changed existing test code to not use the validator,
since we don't need any manual validation of our test cases, and `123`
is not a meaningful timestamp in any case :)

Test plan: `yarn test`
2020-05-13 11:44:23 -07:00
Dandelion Mané
96c34e3c6c
Enable address to alias conversion (#1784)
This commit updates the alias module so that we may convert node
addresses into aliases. Naturally, the node address needs to be some
kind of user node address that is known to the aliasing scheme.

It's a big in-elegant that this creates a "hidden" integration point for
plugins, where plugins creating new user node types should add hardcoded
logic into the identity plugin's alias system. However, it is a
convenience and we currently use this system, so I'm just going to add
this functionality for now, and think about how the alias system should
work long term (or whether we should phase it out) for another
discussion.

This is needed for #1773.

I have `toAlias` return `null` when the address doesn't correspond to a
known aliasing scheme, rather than erroring. I think erroring would be
too harsh, given that it's quite possible that the user has loaded
third-party plugins that haven't registered aliasing schemes upstream
with us. In that case, the application should make a best effort attempt
to proceed without an alias (e.g. fallback to the full address), for
robustness.

Test plan: Unit tests included; `yarn test` passes.
2020-05-13 11:43:01 -07:00
Hammad Jutt
f183c51159
Add grain allocation module (#1743)
* Add grain allocation module

This adds the `grain/allocation.js` module, which contains logic for computing "grain allocations" based on cred scores.

The resultant Allocation type contains the receipts (who gets how much Grain) and the
strategy informing the payout. We can then include these directly into the ledger
to keep a record of the grain balances.

The code is pretty well tested, hitting a number of potential edge cases in the
distribution logic.

Test plan: Inspect attached unit tests, verify CI passes.

Paired with @decentralion

* Rename "Lifetime" distribution strategy to "Balanced" distribution strategy

Balanced more accurately describes what the strategy is doing since its optimizing
to reduce the "imbalance" of cred scores with total grain earnings.

Test Plan: grep "lifetime" to ensure its not used to refer to a distribution strategy

* Refactor strategy specific logic in distribution function

Cleans ups the distribution function by moving all the logic for transforming data and handling errors
into the respective functions (computeImmediateReceipts and computeBalancedReceipts). Also added a test
to ensure error is thrown on an unsupported strategy with an empty credHistory.

Test Plan: Ensure unit tests pass

* Return an empty distribution if budget is zero

Adds a check for when budget === ZERO to return an empty array of receipts to prevent
having receipts with zero grain

Test Plan: Ensure unit tests pass

* Rename earnings to lifetimeEarningsMap

This makes it more descriptive and explicit that the parameter is supposed to be
the lifetime earnings of each contributor, vs "earnings" which could mean many things

Test Plan: Grep "earnings" (case sensitive) to ensure its no longer used in places
where its representing lifetime earnings of all contributors

* Rename distribution to createGrainDistribution

Also renamed the related types / vars from "Distribution" to "GrainDistribution".
This makes it more descriptive and explicit that it is a distribution of Grain,
not a probability distribution or any other distribution. Prevents potential naming
conflicts / confusion with the core/algorithm/distribution module.

Test Plan: Grep "distribution" to ensure that the grain related code always uses
"GrainDistribution"

* Rename GrainDistribution to GrainAllocation

This simplifies the scope of the grain distribution module
to focus purely on the calculations for the Immediate and
Balanced allocation strategies instead of trying to solve
for the actual "ledger of events". The time-filtering has
been removed with the responsibility being delegated to the
caller of the functions. The actual "GrainDistributed" events
can contain the timestamp and an array of "GrainAllocation"s.

Test Plan: Grep "distribution" to ensure its no longer used
in this PR when referring to a GrainAllocation. Also ensure
all the unit tests pass.

* Move "budget" field from Strategy to GrainAllocation

It feels more appropriate for the budget to be a property
of a GrainAllocation since we are allocating that amount of
grain based on a certain strategy. The strategy type is just
meant to describe how the grain was allocated, not how much
grain was allocated. Its also more appropriate since a "distribution"
will have an array of GrainAllocation's instead of GrainAllocation
having an array of strategies, each with different amounts
of grain budget.

Test Plan: Ensure there are no type errors and that Unit tests are passing

* Simplify test cases for GrainAllocation strategies

There was a lot of duplicated test code for the different
allocation strategies, this consolidates it by using describe.each
to run the same test suites on both strategies.

Test Plan: Ensure that unit tests pass and that both strategies
have test coverage for the common test cases

* Rename "lifetimeEarningsMap" to "lifetimeGrainAllocation" and "credMap" to "immediate/lifetimeCredMap"

Changes the naming of some params / variables to be more
descriptive / accurate.

Test Plan: grep "lifetimeEarnings" to ensure its no longer used anywhere

* Improve clarity for balanced allocation test cases

Adds comments to explain the reasoning behind expected receipts
and update calculations to consistently use the "BUDGET"
variable

Test Plan: Ensure CI tests pass
2020-05-11 19:02:08 -06:00
Robin van Boven
338ab3b016
Release: bump version to 0.5.0 (#1787)
See tracking issue #1679 for release notes.
2020-05-08 16:31:26 +02:00
Robin van Boven
a6d184ad23
Chore: upgrade flow-bin (#1786)
Versions >= v0.121.0 require more specific suppression comments.
See https://github.com/facebook/flow/releases/tag/v0.121.0

Closes #1711
2020-05-08 15:31:29 +02:00
Dandelion Mané
7b847120bb
Add sourcecred output command (#1783)
This command is basically a fork of `cli/scores`, except it outputs the
format described in #1773. I started by copying cli/scores and
sharness/test_cli_scores.t, and made appropraite modifications.

You can check out the example-github-output.json to get a feel for the
new format. I also added a compat header in `analysis/output.js`, and
made the necessary adjustments to the CLI harness.

Test plan: The sharness test runs the real command and saves output in
its success case, looking at that JSON is sufficient. I also manually
ran it on the @sourcecred project.
2020-05-07 20:25:52 -07:00
Dandelion Mané
610b9c4827
Compute the V1 output format from TimelineCred (#1782)
This commit builds on #1781, adding the logic for computing the first
output format from TimelineCred. See #1773 for context.

Test plan:
The logic is simple, but has a couple interesting edge cases. I've added
unit tests to cover them. `yarn test` passes.
2020-05-07 20:15:46 -07:00
Robin van Boven
b5fa5abb49
Chore: upgrade packages (#1785)
This includes packages that can be upgraded without making changes.

* Chore: minor version upgrades
* Chore: upgrade fs-extra
* Chore: upgrade chalk
* Chore: upgrade jest, babel-jest

Closes #1712
Closes #1734
2020-05-07 18:48:52 +02:00
Dandelion Mané
b769c92550
Add output data formats (#1781)
As explored in #1773, this commit adds some output data formats that we
can use to enable data analysis of cred instances, along with powering
new UIs. I've included three output formats in order of increasing
complexity, with the intention that we can ship the simpler output
formats quickly, and then enhance with more information to power better
analyses.

I've tried to ensure forward-compatibility with CredRank, so that we can
migrate to CredRank without needing to make major changes to this API.

I may want to include type information in the output format as well.

Test plan: Human review and consideration. `yarn test` passes.

Thanks to @s-ben and @Beanow for discussion, review, and inspiration.
2020-05-07 09:26:02 -07:00
Brian Litwin
286fcb98f4
Discord: create snapshots from test instance (#1736)
This adds a bash script that fetches data from our test
Discord instance. We will be able to test against this
data and easily update the data if Discord's api changes.

Test plan:

After running bash script, inspected snapshot files
and verified that the data appears reasonable.

Verified that the check for `jq` and `SOURCECRED_DISCORD_TOKEN` both
fail and exit if `jq` isn't installed or the Discord bot token
hasn't been set.
2020-05-04 20:38:45 +02:00
Robin van Boven
3748f1862b
Initiatives: add EdgeSpec support to createGraph (#1767)
As of this commit the plugin should fully support EdgeSpec. Meaning the
entries are included in the Graph and Cred computations.

As the champions field does support URLs but not NodeEntry from an
EdgeSpec, we're separating the URL handler from the EdgeSpec handler.
2020-04-27 20:48:48 +02:00
William Chargin
6ec4a3a03c
logo: improve quality of overlapping paths (#1738)
Summary:
This replaces the logo with another SVG document that looks (roughly)
the same but is implemented more cleanly. In particular, the segments of
rays now overlap properly and so are not subject to aliasing, which also
makes it easier to create color variations (e.g., for monochrome).

I would usually optimize this further with SVGO, but this document
appears to reveal a bug in SVGO’s “Round/rewrite paths” optimization
that causes the document to render correctly in Chrome and Firefox but
incorrectly in `rsvg-convert` and Inkscape, so I’m stopping here.

Test Plan:
An animation shows the structure of the new SVG:

![SVG structure animation][gif]

[gif]: https://user-images.githubusercontent.com/4317806/78756263-5da8f880-792f-11ea-9fd1-8e1380e3c530.gif

wchargin-branch: logo-overlapping-paths
2020-04-27 10:50:03 -07:00
Brian Litwin
a1aea75845
Discord: fix whitespace in comments (#1747)
Removes extraneous whitespace in comments in the Discord Plugin.

Test Plan:

Grepping `/*` in the `src/plugins/discord` directory should show
6 instances of this comment style, all of which should have no
whitespace after the final line of content.

`yarn test` passes.
2020-04-27 18:54:38 +02:00
Robin van Boven
9e407b3cac
Initiatives: add EdgeSpec support to Initiative type (#1764)
Because the `Initiative` type now supports `EdgeSpec`, we're no longer
discarding entries when converting between `InitiativeFile` and
`Initiative`.

To make this commit smaller and easier to review, we're not yet adding
support to add `NodeEntry` to the graph though, instead
`createWeightedGraph` ignores the entries for now.
2020-04-27 18:31:10 +02:00
Robin van Boven
0a671025d7
Initiatives: add EdgeSpecJson support to InitativeFile v0.2.0 (#1763)
An additional change here is we're allowing more keys to be omitted in
the JSON format. This is both intuitive for data entry, and safer in
terms of Flow types (as JSON.parse returns any).

The test examples now cover a v0.1.0 (initiative-A), v0.2.0 with just
URLs (initiative-B) and one with just entries (initiative-C).

To make this commit smaller and easier to review, we're not yet adding
`EdgeSpec` to the `Initiative` type and will ignore the entries when
converting from `InitiativeFile` to `Initiative`.
2020-04-27 18:24:07 +02:00
Robin van Boven
5ef69bef50
Initiatives: add addressForNodeEntry (#1758)
Defines the NodeAddress format we want to use for a NodeEntry.

Because we want to use the parent InitiativeId as parts of the address,
we'll need to read the underlying string[], changing the opaque type.
2020-04-27 18:15:18 +02:00
Dandelion Mané
203b48066c
resolveAlias: add support for sourcecred identity (#1752)
This commit modifies the resolveAlias function in the identity plugin's
alias module so that it now allows you to convert sourcecred identity
aliases (e.g. "sourcecred/user") back into NodeAddresses. This will be
necessary so that we can convert our ad-hoc distributions and transfers
(which use aliases, including SourceCred identity aliases) into the more
robust formats for the productionized grain ledger, which use full node
addresses.

I did a slight refactor on the identity module to expose a method for
constructing the address without the rest of the node.

Test plan: `yarn test`
2020-04-26 13:41:58 -07:00
William Chargin
830c83bedf
mirror: replace transaction helper with builtin (#1771)
Summary:
Version 5.0.0 of `better-sqlite3` redesigned the `Database.transaction`
method to do exactly what we want, obviating the need for our custom
helper function and tests. It’s actually more flexible: it allows the
transaction function to take arguments, and allows nested transactions.

Test Plan:
All tests pass as written. If this patch is instead made by changing the
implementation of `_inTransaction` to `return db.transaction(fn)()`,
most tests still pass, excepting only some tests about the behavior of
`_inTransaction` itself around transaction nesting and manual rollbacks;
those behavior changes are acceptable, especially given that they’re not
exercised in the code under test.

wchargin-branch: mirror-builtin-transaction
2020-04-26 13:13:53 -07:00
William Chargin
1f8925fe77
flow: update better-sqlite3 types for v7.0.0 (#1770)
Summary:
I first wrote these type definitions for v4.x.x. The library API has
changed since then. This patch updates the type definitions to match the
current API as of the v7.0.0 release:
<https://github.com/JoshuaWise/better-sqlite3/blob/v7.0.0/docs/api.md>

There are breaking changes, but none among functions that we use. On the
other hand, there are new features that will be useful to us.

Test Plan:
Running `yarn flow` still passes. There may be some errors among typing
of functions that we don’t actually use (particularly `aggregate`, which
is more complicated than the others). If so, we can cross those bridges
when we come to them.

wchargin-branch: flow-better-sqlite3-v7-api
2020-04-26 13:08:51 -07:00
William Chargin
c18244bf19
deps: upgrade better-sqlite3@^7.0.0 (#1769)
Summary:
Notably, this includes a Jest compatibility improvement:
<https://github.com/JoshuaWise/better-sqlite3/issues/162#issuecomment-619401380>

Test Plan:
Existing unit tests pass.

wchargin-branch: better-sqlite3-v7.0.0
2020-04-26 13:04:31 -07:00