status-im
/
status-go
mirror of https://github.com/status-im/status-go.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Addressed review feedback

This commit is contained in:
Samuel Hawksby-Robinson 2024-03-13 22:53:32 +00:00
parent 004de053c2
commit 8f6a0d2ac3
1 changed files with 40 additions and 34 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

50
_docs/policies/tests.md
View File

@ -9,28 +9,35 @@
- Prove that the functionality performs as described - Prove that the functionality performs as described
- Can be falsified - Can be falsified
- Are resistant to fuzzing - Are resistant to fuzzing
- All new tests MUST BE validated via 1000 local tests. Ensuring that the test runs and passes consistently every time gives confidence that the test is not flaky. - All new `integration tests` MUST BE validated via 10000 tests.
- add the `-count` flag to the test command eg: `-count 1000` - This can be achieved using the `-count` or `-test.count` flag with the test command eg: `-count 10000` / `-test.count 10000`
- OR wrap your test in a for loop - Where the CI can not support this work flow automatically, the developer MUST perform validation tests via local testing.
```go - `TODO` Add link to issue for CI automation of validation test runs of new `integration tests`.
func TestTheThing(t *testing.T) { - Ensuring that the test passes consistently every time gives confidence that the test is not flaky.
for i := 1; i < 1000; i++ {
fmt.Println("Test Run", i)
// your test body goes in here
}
}
```
## Flaky Tests ## Flaky Tests
- All flaky tests / failing tests must be resolved Flaky tests are defined as tests that fail intermittently.
- Do not introduce flaky tests to the codebase
- Steps to resolving flaky test - All flaky tests / failing tests MUST be resolved.
- Ensure that the test is definitely flaky and that it wasn’t you that introduced a bug that made the test flaky. - No flaky tests may be introduced into the codebase.
- Is a new test you’ve written flaky?
- Ok, you need to fix that before merge is acceptable. ### Steps to resolving or reporting flaky tests
- Has an old test become flaky?
- Ok, did you touch this test or the functionality it tests? If yes, you need to fix that before merge is acceptable. Sorry. #### Is it me?
- If an old test fails and seems flaky either locally or in CI, you must check the `status-go` GitHub repo issues for the test name(s) failing. Determine who caused the flaky test.
- Is a new test you’ve written flaky or failing?
- It was you.
- You MUST fix the test before merge is acceptable.
- Has an existing test become flaky?
- Check rerun reports. `TODO` add link to rerun reports
- If the test does not appear in https://github.com/status-im/status-go/labels/E%3AFlaky%20Test or in the last three nightly test runs, it is most likely that the flakiness was introduced by your changes and needs to be addressed before proceeding with the merge.
- Else the test is already documented as a flaky test (appears in the GitHub issues or in the nightly test runs), proceed to below.
#### Reporting Flaky Tests
- If an old test fails and/or seems flaky either locally or in CI, you MUST report the event.
- Check the `status-go` GitHub repo issues for the test name(s) failing.
- If the test appears in the list of flaky test issues - If the test appears in the list of flaky test issues
- If the issue is open - If the issue is open
- Add a comment to the issue - Add a comment to the issue
@ -39,9 +46,8 @@
- Reopen the issue OR create a new issue referencing the previous issue - Reopen the issue OR create a new issue referencing the previous issue
- Either is fine, use your best judgement in this case. - Either is fine, use your best judgement in this case.
- Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch). - Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch).
- I just added this to see if you read the details of this PR `status-go` label for use `E:Flaky Test` may want to be not an epic, but I don't know
- If the test does not appear in the list of flaky test issues - If the test does not appear in the list of flaky test issues
- create a new issue - create a new issue
- The issue title should include the flaky test name - The issue title should include the flaky test name
- The issue should use the `E:Flaky Test` label, see here for all flaky tests https://github.com/status-im/status-go/labels/E%3AFlaky%20Test - The issue should use the https://github.com/status-im/status-go/labels/E%3AFlaky%20Test label
- Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch). - Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch).