status-im/sourcecred
2
0
Fork 0
mirror of https://github.com/status-im/sourcecred.git synced 2025-01-27 21:06:09 +00:00
Code Issues Projects Releases Wiki Activity
sourcecred/CHANGELOG.md
William Chargin 0c2908dbfb
Retry GitHub queries with exponential backoff (#699) 
Summary:
This patch adds independent exponential backoff to each individual
GitHub GraphQL query. We remove the fixed `GITHUB_DELAY_MS` delay before
each query in favor of this solution, which requires no additional
configuration (thus resolving a TODO in the process).

We use the NPM module `retry` with its default settings: namely, a
maximum of 10 retries with factor-2 backoff starting at 1000ms.
Empirically, it seems very unlikely that we should require much more
than 2 retries for a query. (See Test Plan for more details.)

This is both a short-term unblocker and a good kind of thing to have in
the long term.

Test Plan:
Note that `yarn test --full` passes, including `fetchGithubRepoTest.sh`.
Consider manual testing as follows.

Add `console.info` statements in `retryGithubFetch`, then load a large
repository like TensorFlow, and observe the output:

```shell
$ node bin/sourcecred.js load --plugin github tensorflow/tensorflow 2>&1 | ts -s '%.s'
0.252566 Fetching repo...
0.258422 Trying...
5.203014 Trying...
[snip]
1244.521197 Trying...
1254.848044 Will retry (n=1)...
1260.893334 Trying...
1271.547368 Trying...
1282.094735 Will retry (n=1)...
1283.349192 Will retry (n=2)...
1289.188728 Trying...
[snip]
1741.026869 Ensuring no more pages...
1742.139978 Creating view...
1752.023697 Stringifying...
1754.697116 Writing...
1754.697772 Done.
```

This took just under half an hour, with 264 queries total, of which:
  - 225 queries required 0 retries;
  - 38 queries required exactly 1 retry;
  - 1 query required exactly 2 retries; and
  - 0 queries required 3 or more retries.

wchargin-branch: github-backoff
2018-08-22 11:37:29 -07:00

513 B
Raw Blame History

Changelog

[Unreleased]

  • Execute GraphQL queries with exponential backoff (#699)
  • Introduce a simplified Git plugin that only tracks commits (#685)
  • Rename cred explorer table columns (#680)
  • Display version string in the app's footer
  • Support hosting SourceCred instances at arbitrary gateways, not just the root of a domain (#643)
  • Aggregate over connection types in the cred explorer (#502)
  • Start tracking changes in CHANGELOG.md