diff --git a/doc/pipeline_process.md b/doc/pipeline_process.md index 858d3d6645..a71c9e9a9a 100644 --- a/doc/pipeline_process.md +++ b/doc/pipeline_process.md @@ -2,7 +2,7 @@ [Pipeline for QA](https://github.com/status-im/status-mobile/projects/7) is a project board for developers and testers used to track the status of a pull request, get reviews and manual testing, and run autotests. -The generally accepted recommendations for its use are described below: +The generally accepted requirements for its use are described below: ## Opening a PR - Once a PR is created, it moves to the ```REVIEW``` column where a review will be requested automatically. @@ -13,35 +13,70 @@ The generally accepted recommendations for its use are described below: ### What if the work is still in progress? - If PR work is not finished yet, please mark it as a draft or add [WIP] to the title and keep it in the `CONTRIBUTOR` column until it's ready to be reviewed/tested. +- Alternatively, you can open a `draft`. +### When is a PR considered ready to move forward? -### When is a PR considered to be Ready for testing by QA team? 

 -Ready for testing PR should meet the following criteria: +Ready for testing, a PR should meet the following criteria: -1. Reviewed and has at least 1 approval -2. Rebased to `develop` branch (both `status-mobile` and `status-go` if needed, depending on what part has changes) -3. All possible conflicts have been resolved -4. Has the label: `request-manual-qa` +1. Reviewed and has at least 1 approval. +2. Rebased to the `develop` branch (both `status-mobile` and `status-go` if needed, depending on what part has changes). + - **NOTE:** QAs can use the `Update with rebase` button from GitHub if: + - The PR has no `status-go` changes. + - The PR has no conflict with the develop branch. +3. All possible conflicts have been resolved. +4. Has the label: `request-manual-qa` or `skip-manual-qa`. +5. PRs **MUST** identify what area is affected and should have a description. +**NOTE:** Make sure that QAs are OK with that + +### Adding `skip-manual-qa` + +- Please ask another team member before adding the `skip-manual-qa` label (PR/Status community/DMs) so that there's a second opinion. +- The PR MUST have a proper reasoning why manual QA is skipped. +- The PR MUST include the steps of testing that has been done by the developer prior to moving it forward. **From the perspective of a developer it means that once work on PR is finished:** 1. It should be rebased to the latest `develop`. If there are conflicts - they should be resolved if possible. 2. If the PR was in the `Contributor` column - it should be moved to `Review` column. 3. Wait for the review. 4. Make sure that after review and before requesting manual QA your PR is rebased to current develop. -5. Once the PR has been approved by reviewer(s) - label `request-manual-qa` should be applied to the PR -6. Move PR to the E2E column when it is ready for testing (**mandatory for all PRs**). That will also trigger e2e tests run. QAs are monitoring PRs from E2E column and take it into test. +5. The PR **MUST** be moved to the E2E column when it is ready for testing (**mandatory for all PRs**). +That will also trigger e2e tests run. QAs are monitoring PRs from E2E column and take it into test. -After that - PR will be taken into manual testing by the QA team. +6. After that - PR will be taken into manual testing by the QA team. + +### E2E tests and analyzing the results + +This step cannot be skipped. So, at least one comment from the `status-im-auto` bot with results is a prerequisite for moving forward. +Information on how to analyze tests can be found [here](https://github.com/status-im/status-mobile/blob/develop/doc/how-to-launch-e2e.md). +Tests might be flaky, as they depend on infrastructure - SauceLabs and Waku. + +If there are `Failed tests` and you are not sure about the reason, you can always ping the mobile QAs for help (preferably in PRs by `@status-im/mobile-qa`). + +**For now, it is the only reliable source of regression in the app because, due to time constraints, it is not possible to test thoroughly each PR. +Please, respect this rule.** ## Testing PR ### Manual testing -- If you think PR needs and is ready for manual testing, please add the ```request-manual-qa``` label. + +#### Prerequisites for manual testing +1. PR has a description of the change and details the area of the application affected by the change. +2. PR has automation results from the previous E2E test run step. +3. PR includes a demo, test steps, and any possible regression(s) +4. If appropriate, PR details what elements of a feature are out of scope. + +**If one of the prerequisites is missing, the PR will be moved to the contributor folder with a corresponding comment.** + +- If you think a PR needs and is ready for manual testing, please add the `request-manual-qa` label. - QA engineer picks up one of PRs with the ```request-manual-qa``` label, drags the item to the ```IN TESTING``` column and assigns it to themselves. - During testing, QA will add comments describing the issues found, and also review automation tests results. Usually found issues are numbered as "Issue 1, Issue 2", etc. -When the first round of testing is completed and all issues for this stage are found, the QA can add the ```Tested - Issues``` label and drag the card to the ```CONTRIBUTOR``` column. These two actions are optional. +When the first round of testing is completed and all issues for this stage are found, the QA can add the ```Tested - Issues``` label and drag the card to the ```CONTRIBUTOR``` column. These two actions are optional. + +**IMPORTANT NOTE:** when the issues are fixed, developer **MUST** notify the QA that it is ready to be re-tested again by mention them in the PR. + - When manual testing of the PR is fully completed and all the issues are fixed, the QA adds the ```Tested - OK``` label and drags the card to the ```Design review``` column (the cases when design review is necessary are described below). - If design review is not required, the QA drags the PR to the ```MERGE``` column. After that the developer merges PR into develop. - If design review has been done, the designer (```@Francesca-G```) drags the PR to the ```MERGE``` column. @@ -68,7 +103,7 @@ There are three possible scenarios when the design review is completed: --- #### Why my PR is in `Contributor` column? -PR can be moved to this column by the ```status-github-bot``` or by QA engineer with label `Tested-issues`. +PR can be moved to this column by the ```status-github-bot``` or by QA engineer with label `Tested-issues` or if one of the requirements for manual QA was not met. In the first case most often this happens due to conflicting files in PR. In the second case - after fixing of all found issues, the developer should ping the QA in the PR comments for retesting. @@ -86,7 +121,6 @@ In the second case - after fixing of all found issues, the developer should ping 6. In case of manual testing - the label ```Tested - OK``` from QA 7. In case of design review - the approval from the designer - You can merge your PR into develop - some useful clues you can find [here](https://notes.status.im/setup-e2e#3-Merging-PR) HAPPY DEVELOPMENT! :tada: