chore(docs): Document some of our existing testing practices (#20691)
Document some of our current testing practices in hopes of helping reduce friction in PRs and communication in general. In theory, nothing in the text should be a surprise because these are things we have been discussing over many months (some things for almost 1.5 years) and are already present in the code.
This commit is contained in:
Icaro Motta
GitHub
parent
e4639c153f
commit
4b8a612df4
|
|
@ -1,67 +1,43 @@
|
|||
## Getting Started
|
||||
|
||||
[Starting Guide](starting-guide.md)
|
||||
|
||||
[IDE Setup](ide-setup.md)
|
||||
|
||||
|
||||
- [Starting Guide](starting-guide.md)
|
||||
- [IDE Setup](ide-setup.md)
|
||||
|
||||
## Development Process
|
||||
|
||||
[Coding guidelines](new-guidelines.md)
|
||||
|
||||
[UI components coding guidelines](ui-guidelines.md)
|
||||
|
||||
[Release Checklist](release-checklist.md)
|
||||
|
||||
[Release Guide](release-guide.md)
|
||||
|
||||
[Merging PR process](merging-pr-process.md)
|
||||
|
||||
[PR Review Policy](pr-review-policy.md)
|
||||
|
||||
[Working on PR together with QA team](pipeline_process.md)
|
||||
|
||||
[Debugging](debugging.md)
|
||||
|
||||
[Patching](patching.md)
|
||||
|
||||
[Creating a pixel perfect UI](pixel-perfection.md)
|
||||
|
||||
[Contributing to status-go](status-go-changes.md)
|
||||
|
||||
[Malli schemas (recorded demo)](https://www.youtube.com/watch?v=SlRio70aYVI) ([slides](files/forging-code-with-schemas-sep-2023-slides.pdf))
|
||||
- [Coding guidelines](new-guidelines.md)
|
||||
- [UI components coding guidelines](ui-guidelines.md)
|
||||
- [Release Checklist](release-checklist.md)
|
||||
- [Release Guide](release-guide.md)
|
||||
- [Merging PR process](merging-pr-process.md)
|
||||
- [PR Review Policy](pr-review-policy.md)
|
||||
- [Working on PR together with QA team](pipeline_process.md)
|
||||
- [Debugging](debugging.md)
|
||||
- [Patching](patching.md)
|
||||
- [Creating a pixel perfect UI](pixel-perfection.md)
|
||||
- [Contributing to status-go](status-go-changes.md)
|
||||
- [Malli schemas (recorded demo)](https://www.youtube.com/watch?v=SlRio70aYVI) ([slides](files/forging-code-with-schemas-sep-2023-slides.pdf))
|
||||
|
||||
## Testing
|
||||
|
||||
[How to run local tests](testing.md)
|
||||
|
||||
[End-to-end tests (e2e) overview](how-to-launch-e2e.md)
|
||||
|
||||
[Component tests (jest) overview](component-tests-overview.md)
|
||||
|
||||
- [Overview](tests/tests-overview.md)
|
||||
- [How to run local tests](tests/how-to-run-local-tests.md)
|
||||
- [End-to-end tests (e2e) overview](tests/how-to-launch-e2e.md)
|
||||
- [Component tests (jest) overview](tests/component-tests-overview.md)
|
||||
|
||||
## Project details
|
||||
|
||||
[status-go introduction (recorded meeting)](https://drive.google.com/file/d/1B7TljmTZ8fHkqJH8ChU1Cp4FGDFM03gq/view)
|
||||
|
||||
[re-frame usage (recorded meeting)](https://drive.google.com/file/d/1qv_E0CEGzQpu_zGXD0gCTU5EvhC2k8Jy/view)
|
||||
|
||||
[status app functionality demo](https://drive.google.com/file/u/0/d/1PmwsMLTLDNNIdv5_6wvMOwoj2PfT50c6/view)
|
||||
- [status-go introduction (recorded meeting)](https://drive.google.com/file/d/1B7TljmTZ8fHkqJH8ChU1Cp4FGDFM03gq/view)
|
||||
- [re-frame usage (recorded meeting)](https://drive.google.com/file/d/1qv_E0CEGzQpu_zGXD0gCTU5EvhC2k8Jy/view)
|
||||
- [status app functionality demo](https://drive.google.com/file/u/0/d/1PmwsMLTLDNNIdv5_6wvMOwoj2PfT50c6/view)
|
||||
|
||||
## Misc
|
||||
|
||||
[Importing icons from Figma into project](export-icons.md)
|
||||
|
||||
[Updating Status APK builds for the F-Droid Android application catalogue](fdroid.md)
|
||||
|
||||
[Troubleshooting for known errors](troubleshooting.md)
|
||||
|
||||
|
||||
- [Importing icons from Figma into project](export-icons.md)
|
||||
- [Updating Status APK builds for the F-Droid Android application catalogue](fdroid.md)
|
||||
- [Troubleshooting for known errors](troubleshooting.md)
|
||||
|
||||
## Outdated:
|
||||
|
||||
[Old guidelines](codebase-structure-and-guidelines.md)
|
||||
|
||||
[Post mortem analysis](post-mortem.md)
|
||||
|
||||
- [Old guidelines](codebase-structure-and-guidelines.md)
|
||||
- [Post mortem analysis](post-mortem.md)
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ Ready for testing, a PR should meet the following criteria:
|
|||
The PR **MUST** be moved to the E2E column when it is ready for testing (**mandatory for all PRs**).
|
||||
That will also trigger e2e tests run. QAs are monitoring PRs from E2E column and take it into test.
|
||||
This step cannot be skipped. So, at least one comment from the `status-im-auto` bot with results is a prerequisite for moving forward.
|
||||
Information on how to analyze tests can be found [here](https://github.com/status-im/status-mobile/blob/develop/doc/how-to-launch-e2e.md).
|
||||
Information on how to analyze tests can be found [here](https://github.com/status-im/status-mobile/blob/develop/doc/tests/how-to-launch-e2e.md).
|
||||
Tests might be flaky, as they depend on infrastructure - SauceLabs and Waku.
|
||||
|
||||
If there are `Failed tests` and you are not sure about the reason, you can always ping the mobile QAs for help (preferably in PRs by `@status-im/mobile-qa`).
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ In that case we save results in TestRail as well and push a comment with test re
|
|||
|
||||
For example: https://github.com/status-im/status-react/pull/9147#issuecomment-540008770
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
The test_send_stt_from_wallet opens link in TestRail https://ethstatus.testrail.net/index.php?/tests/view/890885 where performed steps could be found
|
||||
|
|
@ -34,7 +34,7 @@ List of all runs performed by test jobs could be found here https://ethstatus.te
|
|||
**For credentials for TestRail to see results ping Chu in DM**:
|
||||
|
||||
Opening any test run navigates you to list of test cases with results:
|
||||
|
||||
|
||||
|
||||
|
||||
## What about launching e2e manually
|
||||
|
|
@ -54,7 +54,7 @@ Params to specify:
|
|||
- testrail_case_id: here is the list of test cases which you may find in test rail (4-digit value)
|
||||
|
||||
For easier access you can hit `Rerun tests` in GH comment and testrail_case_id/ apk_name/ pr_id will be filled automatically. For making sure that tests are being rerun on most recent e2e build it is recommended to paste link to the last e2e build in apk_name field. The list of PR builds can be found in Jenkins Builds block on PR page.
|
||||
|
||||
|
||||
And then hit ‘Build’.
|
||||
Once the job starts it picks up specified tests, runs them against provided apk and sends results to pull request.
|
||||
|
||||
|
|
@ -80,7 +80,7 @@ Several examples of when test fails to succeed:
|
|||
|
||||
Example: here is the test results https://github.com/status-im/status-react/pull/13015#issuecomment-1016495043 where one test failed.
|
||||
1. Open the test in TestRail and open session recorded for this test in SauceLabs
|
||||
|
||||
|
||||
|
||||
|
||||
In TestRail you may find all the steps performed by the test.
|
||||
|
|
@ -0,0 +1,167 @@
|
|||
# Tests
|
||||
|
||||
## Introduction
|
||||
|
||||
This document provides a general overview of the types of tests we use and when
|
||||
to use them. It is not meant to be a tutorial or a detailed documentation about
|
||||
testing in software development.
|
||||
|
||||
## Types of tests
|
||||
|
||||
Tests in `status-mobile` are comprised of:
|
||||
|
||||
- Unit tests
|
||||
- Subscription tests
|
||||
- Event tests
|
||||
- Tests for various utilities
|
||||
- [Component tests](./component-tests-overview.md)
|
||||
- Integration/contract tests
|
||||
- [End-to-end tests](./how-to-launch-e2e.md)
|
||||
|
||||
We apply the [test
|
||||
pyramid](https://en.wikipedia.org/wiki/Test_automation#Testing_at_different_levels)
|
||||
strategy, which means we want the majority of tests at the bottom of the
|
||||
pyramid. Those should be fast and deterministic and support REPL-Driven
|
||||
development (RDD). Slightly above them, we have component tests, then
|
||||
integration/contract tests and finally end-to-end tests. The closer to the top
|
||||
of the pyramid, the more valuable a test can be, but also more difficult to
|
||||
pinpoint why it failed and harder to make it dependable.
|
||||
|
||||
*Note*: there are literally dozens of [types of
|
||||
tests](https://en.wikipedia.org/wiki/Software_testing), each with its strengths
|
||||
and weaknesses.
|
||||
|
||||
We tend not to stub or mock implementations in our tests, which means our tests
|
||||
are [sociable](https://martinfowler.com/bliki/UnitTest.html).
|
||||
|
||||
## What to test?
|
||||
|
||||
The UI is driven by global & local state changes caused by events. Global state
|
||||
is managed by re-frame and local state by Reagent atoms or React hooks. Except
|
||||
for component and end-to-end tests, we test only non-UI code in `status-mobile`.
|
||||
Given that the UI is greatly derived from global state, by guaranteeing the
|
||||
state is correct we can prevent bugs and, more importantly, reduce the [cost of
|
||||
change](https://www.pmi.org/disciplined-agile/agile/costofchange).
|
||||
|
||||
We strive to minimize the amount of _business logic_ in views (UI code). We
|
||||
achieve this by moving capabilities to status-go and also by adhering to
|
||||
re-frame's architecture.
|
||||
|
||||
Whenever appropriate (see section `When to test?`), we _may_ test:
|
||||
|
||||
- Re-frame events.
|
||||
- Re-frame subscriptions.
|
||||
- Utility functions.
|
||||
- User journeys through integration/contract tests.
|
||||
|
||||
Interestingly, we don't test re-frame _effects_ in isolation.
|
||||
|
||||
### What are status-mobile integration and contract tests?
|
||||
|
||||
The mobile _integration tests_ can be used to "simulate" user interactions and
|
||||
make actual calls to status-go via the RPC layer and actually receive signals.
|
||||
We can also use these tests to verify the app-db and multiple subscriptions are
|
||||
correct. We use the word _simulate_ because there is no UI. Basically, any flow
|
||||
that can be driven by re-frame events is possible to automatically test. There
|
||||
is no way to change or inspect local state managed by React.
|
||||
|
||||
A _contract test_ has the same capabilities as an integration test, but we want
|
||||
to draw the line that they should focus more on a particupar RPC endpoint or
|
||||
signal, and not on a user journey (e.g. create a wallet account). In the future,
|
||||
we may consider running them automatically in status-go.
|
||||
|
||||
**Note:** integration tests and contract tests are currently overlapping in
|
||||
their responsibilities and still require a clearer distinction.
|
||||
|
||||
## When to test?
|
||||
|
||||
(Automated) tests basically exist to support rapid software changes, but not
|
||||
every piece of code should be tested. The following are general recommendations,
|
||||
not rules.
|
||||
|
||||
- What would be the consequences to the user of a bug in the implementation you
|
||||
are working on?
|
||||
- Can a QA exercise all the branches in the code you changed? Not surprisingly,
|
||||
usually QAs can't test many code paths (it may be nearly impossible), and
|
||||
because PRs are not often tested by reviewers, many PRs can get into `develop`
|
||||
without the necessary quality assurance.
|
||||
- How costly was it for you to verify a function/event/etc was correct? Now
|
||||
consider that this cost will be dispersed to every developer who needs to
|
||||
change the implementation if there are no tests.
|
||||
- Check the number of conditionals, and if they nest as well. Every conditional
|
||||
may require two different assertions, and the number of assertions can grow
|
||||
exponentially.
|
||||
- How complicated are the arguments to the function? If they contain nested maps
|
||||
or data that went through a few transformations, it may be tricky to decipher
|
||||
what they are, unless you are familiar with the code. A test would be able to
|
||||
capture the data, however complex they are.
|
||||
|
||||
### When to unit-test subscriptions?
|
||||
|
||||
Only test [layer-3
|
||||
subscriptions](https://day8.github.io/re-frame/subscriptions/#the-four-layers),
|
||||
i.e. don't bother testing extractor subscriptions (check the related
|
||||
[guideline](https://github.com/status-im/status-mobile/blob/7774c4eac16fdee950a17bf5d07630c45a980f41/doc/new-guidelines.md#subscription-tests)).
|
||||
Some layer-3 subscriptions can still be straightforward and may not be worth
|
||||
testing.
|
||||
|
||||
- Check the number of _inputs_ to the sub (from the graph). The higher this
|
||||
number, the greater the chance the subscription can break if any of the
|
||||
input's implementation changes.
|
||||
|
||||
**Note**: if a tested subscription changes inadvertently, even if its own tests
|
||||
still pass, other subscriptions that depend on it and have tests may still fail.
|
||||
This is why we don't directly test the subscription handler, but instead, use
|
||||
the macro `test-helpers.unit/deftest-sub`.
|
||||
|
||||
### When to unit-test events?
|
||||
|
||||
A good hint is to ask if you and other CCs need to rely on re-frisk, UI, REPL,
|
||||
or FlowStorm to understand the event. If the answer is yes or probably, then a
|
||||
test would be prudent.
|
||||
|
||||
- Many events only receive arguments and pass them along without much or any
|
||||
transformation to an RPC call. These are straightforward and usually don't
|
||||
need tests ([example](https://github.com/status-im/status-mobile/blob/7774c4eac16fdee950a17bf5d07630c45a980f41/src/status_im/contexts/contact/blocking/events.cljs#L79-L85)).
|
||||
- Overall, every event basically returns two effects at most, `:fx` and/or
|
||||
`:db`. Usually, the complicated part lies in the computation to return the new
|
||||
app-db. If the event doesn't perform transformations in the app-db or just
|
||||
does a trivial `assoc`, for example, it may not be worth testing.
|
||||
|
||||
For reference, the re-frame author particularly [suggests testing events and
|
||||
subscriptions](https://github.com/day8/re-frame/blob/09e2d7132c479aa43f2a64164e54e42bf8511902/docs/Testing.md#what-to-test).
|
||||
|
||||
### When to unit-test utility functions?
|
||||
|
||||
Most utility functions in `status-mobile` are pure and can be readily and
|
||||
cheaply tested.
|
||||
|
||||
- If the utility is used in an event/subscription and if the event/subscription
|
||||
has tests, you may prefer to test the event/subscription and not the utility,
|
||||
or the other way around sometimes.
|
||||
- If the utility is tricky to verify, such as functions manipulating time, write
|
||||
tests ([example](https://github.com/status-im/status-mobile/blob/7774c4eac16fdee950a17bf5d07630c45a980f41/src/utils/datetime.cljs#L1)).
|
||||
- Utilities can be particularly hard to verify by QAs because they can be lower
|
||||
level and require very particular inputs. In such cases, consider writing
|
||||
tests.
|
||||
|
||||
### When to write integration/contract tests?
|
||||
|
||||
- You want to make real calls to status-go because you think the unit tests are
|
||||
not enough (test pyramid strategy).
|
||||
- You constantly need to retest the same things on the UI, sometimes over
|
||||
multiple screens.
|
||||
- The flow is too important to rely only on manual QA, which can't always be
|
||||
done due to resource limits, so an integration/contract test fills this gap.
|
||||
- You want to rely less on end-to-end tests, which can be more unreliable and
|
||||
slower to change.
|
||||
- You want automatic verifications for some area of the mobile app whenever
|
||||
status-go is upgraded.
|
||||
|
||||
**Note**: the feedback cycle to write integration tests is longer than unit
|
||||
tests because they are slower and harder to debug. Using the REPL with them is
|
||||
difficult due to their stateful nature.
|
||||
|
||||
### When to test Quo components?
|
||||
|
||||
This is covered in [quo/README.md#component-tests](https://github.com/status-im/status-mobile/blob/7774c4eac16fdee950a17bf5d07630c45a980f41/src/quo/README.md#component-tests).
|
||||
Loading…
Reference in New Issue