Skip to content

Commit

Permalink
Fix and improve testing docs (#1792)
Browse files Browse the repository at this point in the history
Improve docs
  • Loading branch information
SamuelWei authored Jan 20, 2025
1 parent d7d971d commit 8a84826
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 45 deletions.
22 changes: 3 additions & 19 deletions docs/docs/development/04-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,26 +27,10 @@ existing issue, the pull request will be checked by a maintainer of this reposit

A new development shouldn't decrease the code testing coverage. Everything in the application must be covered by
appropriate unit and feature tests, depending on the case. In case of bugfixes a test, that fails in the appropriate
case should be implemented, to make regression tests possible for further changes in the application. For the backend the
api can be tested by using feature tests and other functions just with unit tests. For more information about tests
checkout the [Laravel testing guides](https://laravel.com/docs/11.x/testing).
case should be implemented, to make regression tests possible for further changes in the application.

To run the integration tests against a real BigBlueButton server you need to set the test server in
the `.env` file:

```bash
BBB_TEST_SERVER_HOST=https://your-bbb-server.com/bigbluebutton/
BBB_TEST_SERVER_SECRET=your-secret
```

### Frontend

Since the refactoring of the frontend to Vue 3 and PrimeVue, the frontend tests are still a work in progress.
We have started adding some basic [Cypress](https://www.cypress.io) tests, but they are currently incomplete, and code coverage is not yet implemented.
The old Vitest tests have been removed. If you need to reference them, please check the old branches.

If you have experience with Cypress and would like to contribute,
your help is welcome! Please feel free to create a PR.
Please have a look into the [Testing](./07-testing.md) documentation to learn more about the testing strategies
and how to run the tests.

## Submit changes

Expand Down
63 changes: 37 additions & 26 deletions docs/docs/development/07-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,29 @@ title: Testing

Our goal is to build a high-quality product by focusing equally on robust architecture, smooth UI/UX design, and accessible experiences. By combining these elements, we aim to deliver a stable, reliable, and user-friendly platform.

Maintaining product quality is an ongoing process that requires consistent effort. To prevent quality degradation over time, we have established a comprehensive testing strategy, which includes the following components:
Maintaining product quality is an ongoing process that requires consistent effort. To prevent quality degradation over time, we have established a comprehensive testing strategy that includes the following components:

1. **PHP Backend Testing**

- We employ multiple levels of backend tests, including unit tests, integration tests, and more. These tests ensure that each component functions correctly and integrates seamlessly with others.
- We use several levels of backend testing, including unit testing, integration testing and more. These tests ensure that each component works correctly and integrates seamlessly with others.

2. **Frontend Testing**

- Our frontend tests focus on isolated End-to-End (E2E) testing, with most tests stubbing the backend. This allows us to validate the frontend functionality independently and comprehensively.

3. **System Testing**

- We conduct system tests to verify that the entire application operates as expected. These include ensuring the Docker container initializes correctly and confirming that the frontend and backend work together seamlessly.
- We perform system testing to verify that the entire application works as expected. This includes ensuring that the Docker container initialises correctly, and confirming that the frontend and backend work together seamlessly.

4. **Visual Regression Testing**
- Visual regression tests are our latest addition. These tests help us quickly identify and address visual inconsistencies, ensuring a consistent user interface and experience.
- Visual regression testing is our latest addition. These tests help us to quickly identify and fix visual inconsistencies, ensuring a consistent user interface and experience.

By combining these testing methodologies, we ensure that PILOS maintains its high standards of quality and continues to meet user expectations.
By combining these testing methods, we ensure that PILOS maintains its high quality standards and continues to meet user expectations.

## PHP Backend Testing with PHPUnit

The backend of PILOS is written in PHP and uses the Laravel framework.
Laravel provides a robust PHPUnit based test system that allows us to test the application at different levels.
Laravel provides a robust PHPUnit based testing system that allows us to test the application on different levels.
You can learn more about Laravel testing in the [official documentation](https://laravel.com/docs/11.x/testing).

To run the full test suite using a real BigBlueButton server you need to set the test server in the `.env` file:
Expand Down Expand Up @@ -65,30 +65,37 @@ For testing Cypress is used. You can learn more about Cypress in the [official d
The tests stub the backend API requests with fixtures. This allows us to test the frontend in isolation and without the need for a running backend.
It also helps to simulate a lot of different edge cases and error scenarios.

To run the frontend tests, you first have to install cypress outside the container:
### Installing Cypress

First you need to fulfill the [system requirements](https://docs.cypress.io/app/get-started/install-cypress#System-requirements) of Cypress.
Please make sure your OS is supported and all required packages are installed.

If the system requirements are met, you can install Cypress on your local machine:

```bash
npx cypress install
```

Next you need to have the frontend either running compiled (`./sail npm run build`) or in development mode (`./sail npm run dev`).
### Running Cypress Tests

First you need to have the frontend either running compiled (`./sail npm run build`) or in development mode (`./sail npm run dev`).

You can then run the tests in the interactive mode with:

```bash
npx cypress open
npm run cypress:open
```

Or you can run the tests headless with:

```bash
npx cypress run
npm run cypress:run
```

### Coverage

To generate a code coverage report the frontend first has to be instrumented to the coverage can be measured.
This can build the frontend with the coverage instrumentation:
To generate a code coverage report, the frontend must first be instrumented so that the coverage can be measured.
This command can be used to build the frontend with coverage instrumentation:

```bash
./sail npm run build -- --config vite.config.coverage.js
Expand All @@ -97,7 +104,7 @@ This can build the frontend with the coverage instrumentation:
After that you can run the tests with the coverage report:

```bash
npx cypress run
npm run cypress:run
```

To generate a code coverage report in HTML format you can use the following command:
Expand All @@ -109,42 +116,46 @@ To generate a code coverage report in HTML format you can use the following comm
## System Testing with Cypress

System tests are used to verify that the entire application operates as expected.
To ensure that the frontend and backend work together seamlessly, the tests are using the real backend API and a real BigBlueButton server.
The tests are also using Cypress for functional testing.

To run the system tests we have a separate docker-compose file that needs to be started:
To run the system tests a separate docker-compose file needs to be started:

```bash
docker compose -f compose.test.yml -f compose.test.local.yml up -d
```

Before running the tests make sure you have correctly installed cypress as described in the [Installing cypress](#installing-cypress) section.

You can then run the tests in the interactive mode with:

```bash
npx cypress open -P ./tests/System
npm run cypress:open -- -P ./tests/System
```

Or you can run the tests headless with:

```bash
npx cypress run -P ./tests/System
npm run cypress:run -- -P ./tests/System
```

## Visual Regression Testing with Happo.io

We aim to cover as much functionality as possible with our frontend tests. However, testing every aspect of the application can be challenging
and sometimes simply not possible, e.g. layout shifts, svg rendered charts, etc.
We aim to cover as much functionality as possible with our frontend tests. However, testing every aspect of the application can be challenging and sometimes simply not possible, e.g. layout shifts, svg rendered charts, etc.

PILOS relies on several third-party libraries, such as UI Components (PrimeVue), Charts (Chart.js), and others. While these libraries are generally high quality and well-tested, they may occasionally introduce bugs, regressions, or integration issues within our application. Frequent updates to these libraries further necessitate ensuring that our application continues to look and behave as expected.
PILOS relies on several third-party libraries, such as UI components (PrimeVue), charts (Chart.js) and others. While these libraries are generally high quality and well tested, they may occasionally introduce bugs, regressions or integration issues into our application. Frequent updates to these libraries also require us to ensure that our application continues to look and behave as expected.

Identifying visual regressions manually is a time-consuming and error-prone process, which has been a significant pain point in the past. To address this, we have implemented visual regression testing using [Happo](https://happo.io/).
Manually identifying visual regressions is a time-consuming and error-prone process that has been a significant pain point in the past. To address this, we implemented visual regression testing using [Happo](https://happo.io/).

Happo is a powerful tool that enables us to take screenshots of our application and compare them against previous versions to detect visual changes.
Integrating Happo into our existing Cypress test system was straightforward, and we will extend its use to all areas of the application where visual changes are expected.
Happo is a powerful tool that allows us to take screenshots of our application and compare them to previous versions to detect visual changes. Integrating Happo into our existing Cypress test system was straightforward and we will extend its use to all areas of the application where visual changes are expected.

By using Happo for visual regression testing, we can maintain a high standard of visual quality in our application while minimising manual effort.

### Automated Testing Environment

To ensure consistent results, we run visual regression tests exclusively in the GitHub Action runner. This controlled environment minimizes discrepancies and provides reliable feedback.

When a screenshot differs from the baseline, we can review the changes and determine whether it indicate a bug or an intentional update.
If a screenshot differs from the baseline, we can review the changes and determine whether this is a bug or an intentional change.

### Adding Screenshots in Tests

Expand All @@ -158,10 +169,10 @@ cy.get("button").happoScreenshot();

Although visual regression tests run exclusively in the GitHub Action runner, you can write and debug tests locally. Use the following command to open Cypress for interactive debugging:

Before running the tests make sure you have correctly installed cypress as described in the [Installing cypress](#installing-cypress) section.

```bash
npx cypress open -P ./tests/Visual --config experimentalInteractiveRunEvents=true
npm run cypress:open -- -P ./tests/Visual --config experimentalInteractiveRunEvents=true
```

This setup allows you to ensure your tests are functioning correctly before committing changes.

By leveraging Happo for visual regression testing, we can maintain a high standard of visual quality in our application while minimizing manual effort.

0 comments on commit 8a84826

Please sign in to comment.