sourcecred/.circleci/config.yml

142 lines
3.9 KiB
YAML
Raw Normal View History

# JavaScript Node CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-javascript/ for more details
# See https://circleci.com/docs/2.0/config-intro/#section=configuration for spec
#
version: 2.1
orbs:
# https://circleci.com/orbs/registry/orb/circleci/docker
docker: circleci/docker@0.5.13
executors:
node10:
docker:
- image: circleci/node:10
working_directory: ~/repo
node12:
docker:
- image: circleci/node:12
working_directory: ~/repo
commands:
set_up_node_modules:
description: "load node_modules, updating and storing back into cache if necessary"
steps:
- restore_cache:
keys:
- v1-dependencies-{{ checksum "package.json" }}
# fall back to using the latest cache if no exact match is found
- v1-dependencies-
- run: yarn install
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ checksum "package.json" }}
pull_cache:
description: Pulls docker images usable for our cache
steps:
- run: docker pull sourcecred/sourcecred:dev
- run: docker pull node:12
jobs:
test:
executor: node12
steps:
- checkout
- set_up_node_modules
Fix flakey CI memory issues (#1230) Ever since I upgraded all of the dependencies, we've been having regular CI failures, which seem to share a common root cause of memory exhaustion. Here are some examples: [1], [2]. [1]: https://circleci.com/gh/sourcecred/sourcecred/1246 [2]: https://circleci.com/gh/sourcecred/sourcecred/1239 After some experimentation, I've found that we can solve the issue by ensuring that jest runs on its own in CI, so that it doesn't contend with other tests for memories. Also, I reduce its max workers to 2, which matches the number of CPUs in the CircleCI containers. Unfortunately, this does increase our build time. The postcommit (non full) test now takes 45-60s (up from 30-50s), and the full test is also a little slower. However, building in about one minute is still acceptably fast, and having regular flakey test failures is not acceptable, so this is still a win. If we want to improve on this in the future, we should look into the git shells getting spawned in `config/env.js`. I noticed that they were often involved in the out-of-memory failures. Also, I modified `.circleci/config.yml` so that any branch matching the regular expression `/ci-.*/` will trigger a full build. That makes it easier to test against CI failures. Test plan: I ran about ~10 full builds with this change, and more with similar variations, and they all passed. Verify that the full builds that are run for this commit also all pass! Also, verify that running yarn test locally has unchanged behavior, and running `yarn test --ci` locally lets jest run to completion before running any other test.
2019-07-16 00:51:14 +00:00
- run: yarn test --ci
test_full:
executor: node12
steps:
- checkout
- set_up_node_modules
Fix flakey CI memory issues (#1230) Ever since I upgraded all of the dependencies, we've been having regular CI failures, which seem to share a common root cause of memory exhaustion. Here are some examples: [1], [2]. [1]: https://circleci.com/gh/sourcecred/sourcecred/1246 [2]: https://circleci.com/gh/sourcecred/sourcecred/1239 After some experimentation, I've found that we can solve the issue by ensuring that jest runs on its own in CI, so that it doesn't contend with other tests for memories. Also, I reduce its max workers to 2, which matches the number of CPUs in the CircleCI containers. Unfortunately, this does increase our build time. The postcommit (non full) test now takes 45-60s (up from 30-50s), and the full test is also a little slower. However, building in about one minute is still acceptably fast, and having regular flakey test failures is not acceptable, so this is still a win. If we want to improve on this in the future, we should look into the git shells getting spawned in `config/env.js`. I noticed that they were often involved in the out-of-memory failures. Also, I modified `.circleci/config.yml` so that any branch matching the regular expression `/ci-.*/` will trigger a full build. That makes it easier to test against CI failures. Test plan: I ran about ~10 full builds with this change, and more with similar variations, and they all passed. Verify that the full builds that are run for this commit also all pass! Also, verify that running yarn test locally has unchanged behavior, and running `yarn test --ci` locally lets jest run to completion before running any other test.
2019-07-16 00:51:14 +00:00
- run: yarn test --full --ci
test_full_10:
executor: node10
steps:
- checkout
- set_up_node_modules
Fix flakey CI memory issues (#1230) Ever since I upgraded all of the dependencies, we've been having regular CI failures, which seem to share a common root cause of memory exhaustion. Here are some examples: [1], [2]. [1]: https://circleci.com/gh/sourcecred/sourcecred/1246 [2]: https://circleci.com/gh/sourcecred/sourcecred/1239 After some experimentation, I've found that we can solve the issue by ensuring that jest runs on its own in CI, so that it doesn't contend with other tests for memories. Also, I reduce its max workers to 2, which matches the number of CPUs in the CircleCI containers. Unfortunately, this does increase our build time. The postcommit (non full) test now takes 45-60s (up from 30-50s), and the full test is also a little slower. However, building in about one minute is still acceptably fast, and having regular flakey test failures is not acceptable, so this is still a win. If we want to improve on this in the future, we should look into the git shells getting spawned in `config/env.js`. I noticed that they were often involved in the out-of-memory failures. Also, I modified `.circleci/config.yml` so that any branch matching the regular expression `/ci-.*/` will trigger a full build. That makes it easier to test against CI failures. Test plan: I ran about ~10 full builds with this change, and more with similar variations, and they all passed. Verify that the full builds that are run for this commit also all pass! Also, verify that running yarn test locally has unchanged behavior, and running `yarn test --ci` locally lets jest run to completion before running any other test.
2019-07-16 00:51:14 +00:00
- run: yarn test --full --ci
workflows:
version: 2.0
commit:
jobs:
- test
# We also run full-tests on each commit, but they're not configured as
# blocking checks. Thus, in the case that they complete before you want
# to merge your pull request, they provide information, and otherwise
# they don't get in your way.
- test_full
- test_full_10
- docker/publish:
deploy: false
image: sourcecred/sourcecred
tag: latest
requires:
- test
filters:
branches:
ignore:
- master
extra_build_args: --cache-from=node:12,sourcecred/sourcecred:dev
before_build:
- pull_cache
after_build:
- run:
name: Preview Docker Tag for Build
command: |
DOCKER_TAG=$(docker run sourcecred/sourcecred --version | cut -d' ' -f2)
echo "Version that would be used for Docker tag is ${DOCKER_TAG}"
- docker/publish:
image: sourcecred/sourcecred
tag: latest
requires:
- test
filters:
branches:
ignore: /.*/
tags:
only: /^v.*/
extra_build_args: --cache-from=node:12,sourcecred/sourcecred:dev
before_build:
- pull_cache
after_build:
- run:
name: Publish Docker Tag with Sourcecred Version
command: |
DOCKER_TAG=$(docker run sourcecred/sourcecred --version | cut -d' ' -f2)
echo "Version for Docker tag is ${DOCKER_TAG}"
docker tag sourcecred/sourcecred:latest sourcecred/sourcecred:${DOCKER_TAG}
docker push sourcecred/sourcecred
- docker/publish:
image: sourcecred/sourcecred
extra_build_args: --cache-from=node:12,sourcecred/sourcecred:dev
before_build:
- pull_cache
requires:
- test
- test_full
- test_full_10
tag: dev
filters:
branches:
only: master
nightly:
triggers:
- schedule:
cron: "0 22 * * *" # 22:00 UTC
filters:
branches:
only:
- master
jobs:
- test_full
- test_full_10