From 1def7e5505f4f6c2d86ca01efc2966765fd7105b Mon Sep 17 00:00:00 2001 From: Anton Danchenko Date: Wed, 18 Apr 2018 12:45:05 +0300 Subject: [PATCH] initial draft for functional automated testing as a part of Continuous Delivery process (#167) --- README.md | 10 +-- .../README.md | 61 +++++++++++++++++++ 2 files changed, 67 insertions(+), 4 deletions(-) create mode 100644 ideas/167-extended-automated-testing-for-continuous-delivery/README.md diff --git a/README.md b/README.md index 2d8952a..8e19b59 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,7 @@ aborted. | [140-sob-improve-onboarding](ideas/140-sob-improve-onboarding/) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | | [122-sob-metrics](ideas/122-sob-metrics.md) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | | [154-support-web3.js-library](ideas/154-support-web3.js-library.md) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | +| [167-extended-automated-testing-for-continuous-delivery](ideas/167-extended-automated-testing-for-continuous-delivery) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | ## Draft :seedling: and limbo :question: | Idea | State | Success metrics? | Exit criteria? | Clear roles? | Future iteration? | @@ -86,9 +87,10 @@ progress again. This ensures the registry is kept up to date. | @anna | 58, 80, 87, 127 | | @adriacidre | 63, 146 | | @andmironov | 68, 80, 134 | +| @antdanchenko | 167 | | @adambabik | 58, 63, 68, 92 | | @alwx | 134 | -| @asemiankevich | 87, 127, 134 | +| @asemiankevich | 87, 127, 134, 167 | | @cammellos | 87 | | @chadyj | 68, 80, 87, 99, 134 | | @churik |140, 127 | @@ -98,17 +100,17 @@ progress again. This ensures the registry is kept up to date. | @hester | 80, 134 | | @janherich | 87 | | @jeluard | 68, 101 | -| @lukaszfryc | 68, 83 | +| @lukaszfryc | 68, 83, 167 | | @mandrigin | 83 | | @martin | 120 | | @MaxRis | 34 | | @naghdy | 120 | | @oskarth | 120, 121 | -| @serhy | 87 | +| @serhy | 87, 167 | | @themue | 58, 92 | | @vkjr | 34 | | @yenda | 87 | -| @yevh-berdnyk | 80 | +| @yevh-berdnyk | 80, 167 | | @martinklepsch | 140, 121 | | @rcullito | 140 | | @EugeOrtiz | 140 | diff --git a/ideas/167-extended-automated-testing-for-continuous-delivery/README.md b/ideas/167-extended-automated-testing-for-continuous-delivery/README.md new file mode 100644 index 0000000..ec4c35d --- /dev/null +++ b/ideas/167-extended-automated-testing-for-continuous-delivery/README.md @@ -0,0 +1,61 @@ +## Preamble + + Idea: 167 + Title: Functional automated testing as a part of Continuous Delivery process + Status: In Progress + Created: 2018-04-12 + +## Summary +For automated testing to be a part of CD process we need to: +- increase coverage +- increase reliability +- provide visible results on daily basis + +## Swarm Participants +- Lead Contributor: @antdanchenko +- Testing & Evaluation: @antdanchenko +- Contributor: @yevh-berdnyk +- Contributor: @lukaszfryc +- Contributor: @Serhy +- Contributor: @asemiankevich +- PM: + +## Product Overview +Extended set of automated tests which will run against each nightly build in order to provide results via TestRun in TestRail + short summary via Slack notification +As result during nightly build testing a tester should execute only those tests which are not covered by automation in the same TestRun +In case of failure in automated test a tester should add an appropriate status to the test (see `An option to review results should be added for each test via adding 4 statuses to TestRail` in `Product Description` section) +Actual sanity test case which is running against each PR will keep running without changes + + +### Product Description +- Each automated test suite should be based on existing set of cases from TestRail for tracking percents of functional test coverage +- Each automated test should be the reflection of a test case from TestRail +- Results of each automated test from a build should be provided via TestRun in TestRail and linked to the appropriate case +- An option to review results should be added for each test via adding 4 statuses to TestRail: + - feature changed (for changed product behavior) + - known bug (link to the issue) + - false failure (bug in test) + - infrastructure issue + +### Minimum Viable Product +Goal Date: 2018-05-01 + +- New test suite is add to TestRail (`Functional tests for Nightly Build`) +- 80% of test cases for nightly build test suite from TestRail are covered by automated tests +- Extended set of automated tests is running against each nightly build +- Detailed test results are provided in TestRail via TestRuns + short summary via Slack notification + +Testing Days required: 10 + +## Success Metrics +- 80% of functional test cases in `Functional tests for Nightly Build` are covered +- not more than 10% of tests are failing due to an issue with a test +- results are visible and understandable for all status-react team members + +## Exit criteria +Extended set of automated tests covers 80% of functional nightly test cases from TestRail +The set is running against each nightly build +Results are visible in TestRail as TestRuns, also summary is visible in Slack notifications + +## Copyright +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).