mirror of https://github.com/status-im/swarms.git
initial draft for functional automated testing as a part of Continuous Delivery process (#167)
This commit is contained in:
Anton Danchenko
GitHub
parent
7a00005456
commit
1def7e5505
10
README.md
10
README.md
|
|
@ -45,6 +45,7 @@ aborted.
|
|||
| [140-sob-improve-onboarding](ideas/140-sob-improve-onboarding/) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes |
|
||||
| [122-sob-metrics](ideas/122-sob-metrics.md) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes |
|
||||
| [154-support-web3.js-library](ideas/154-support-web3.js-library.md) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes |
|
||||
| [167-extended-automated-testing-for-continuous-delivery](ideas/167-extended-automated-testing-for-continuous-delivery) | :walking_man: In Progress | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes | :white_check_mark: Yes |
|
||||
|
||||
## Draft :seedling: and limbo :question:
|
||||
| Idea | State | Success metrics? | Exit criteria? | Clear roles? | Future iteration? |
|
||||
|
|
@ -86,9 +87,10 @@ progress again. This ensures the registry is kept up to date.
|
|||
| @anna | 58, 80, 87, 127 |
|
||||
| @adriacidre | 63, 146 |
|
||||
| @andmironov | 68, 80, 134 |
|
||||
| @antdanchenko | 167 |
|
||||
| @adambabik | 58, 63, 68, 92 |
|
||||
| @alwx | 134 |
|
||||
| @asemiankevich | 87, 127, 134 |
|
||||
| @asemiankevich | 87, 127, 134, 167 |
|
||||
| @cammellos | 87 |
|
||||
| @chadyj | 68, 80, 87, 99, 134 |
|
||||
| @churik |140, 127 |
|
||||
|
|
@ -98,17 +100,17 @@ progress again. This ensures the registry is kept up to date.
|
|||
| @hester | 80, 134 |
|
||||
| @janherich | 87 |
|
||||
| @jeluard | 68, 101 |
|
||||
| @lukaszfryc | 68, 83 |
|
||||
| @lukaszfryc | 68, 83, 167 |
|
||||
| @mandrigin | 83 |
|
||||
| @martin | 120 |
|
||||
| @MaxRis | 34 |
|
||||
| @naghdy | 120 |
|
||||
| @oskarth | 120, 121 |
|
||||
| @serhy | 87 |
|
||||
| @serhy | 87, 167 |
|
||||
| @themue | 58, 92 |
|
||||
| @vkjr | 34 |
|
||||
| @yenda | 87 |
|
||||
| @yevh-berdnyk | 80 |
|
||||
| @yevh-berdnyk | 80, 167 |
|
||||
| @martinklepsch | 140, 121 |
|
||||
| @rcullito | 140 |
|
||||
| @EugeOrtiz | 140 |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,61 @@
|
|||
## Preamble
|
||||
|
||||
Idea: 167
|
||||
Title: Functional automated testing as a part of Continuous Delivery process
|
||||
Status: In Progress
|
||||
Created: 2018-04-12
|
||||
|
||||
## Summary
|
||||
For automated testing to be a part of CD process we need to:
|
||||
- increase coverage
|
||||
- increase reliability
|
||||
- provide visible results on daily basis
|
||||
|
||||
## Swarm Participants
|
||||
- Lead Contributor: @antdanchenko
|
||||
- Testing & Evaluation: @antdanchenko
|
||||
- Contributor: @yevh-berdnyk
|
||||
- Contributor: @lukaszfryc
|
||||
- Contributor: @Serhy
|
||||
- Contributor: @asemiankevich
|
||||
- PM:
|
||||
|
||||
## Product Overview
|
||||
Extended set of automated tests which will run against each nightly build in order to provide results via TestRun in TestRail + short summary via Slack notification
|
||||
As result during nightly build testing a tester should execute only those tests which are not covered by automation in the same TestRun
|
||||
In case of failure in automated test a tester should add an appropriate status to the test (see `An option to review results should be added for each test via adding 4 statuses to TestRail` in `Product Description` section)
|
||||
Actual sanity test case which is running against each PR will keep running without changes
|
||||
|
||||
|
||||
### Product Description
|
||||
- Each automated test suite should be based on existing set of cases from TestRail for tracking percents of functional test coverage
|
||||
- Each automated test should be the reflection of a test case from TestRail
|
||||
- Results of each automated test from a build should be provided via TestRun in TestRail and linked to the appropriate case
|
||||
- An option to review results should be added for each test via adding 4 statuses to TestRail:
|
||||
- feature changed (for changed product behavior)
|
||||
- known bug (link to the issue)
|
||||
- false failure (bug in test)
|
||||
- infrastructure issue
|
||||
|
||||
### Minimum Viable Product
|
||||
Goal Date: 2018-05-01
|
||||
|
||||
- New test suite is add to TestRail (`Functional tests for Nightly Build`)
|
||||
- 80% of test cases for nightly build test suite from TestRail are covered by automated tests
|
||||
- Extended set of automated tests is running against each nightly build
|
||||
- Detailed test results are provided in TestRail via TestRuns + short summary via Slack notification
|
||||
|
||||
Testing Days required: 10
|
||||
|
||||
## Success Metrics
|
||||
- 80% of functional test cases in `Functional tests for Nightly Build` are covered
|
||||
- not more than 10% of tests are failing due to an issue with a test
|
||||
- results are visible and understandable for all status-react team members
|
||||
|
||||
## Exit criteria
|
||||
Extended set of automated tests covers 80% of functional nightly test cases from TestRail
|
||||
The set is running against each nightly build
|
||||
Results are visible in TestRail as TestRuns, also summary is visible in Slack notifications
|
||||
|
||||
## Copyright
|
||||
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
|
||||
Loading…
Reference in New Issue