From 8a84826d197fd5774a6d24ddaf4664ffc379de8e Mon Sep 17 00:00:00 2001 From: Samuel Weirich <4281791+SamuelWei@users.noreply.github.com> Date: Mon, 20 Jan 2025 09:25:45 +0100 Subject: [PATCH] Fix and improve testing docs (#1792) Improve docs --- docs/docs/development/04-workflow.md | 22 ++-------- docs/docs/development/07-testing.md | 63 ++++++++++++++++------------ 2 files changed, 40 insertions(+), 45 deletions(-) diff --git a/docs/docs/development/04-workflow.md b/docs/docs/development/04-workflow.md index 027f0b5a2..543c24c8f 100644 --- a/docs/docs/development/04-workflow.md +++ b/docs/docs/development/04-workflow.md @@ -27,26 +27,10 @@ existing issue, the pull request will be checked by a maintainer of this reposit A new development shouldn't decrease the code testing coverage. Everything in the application must be covered by appropriate unit and feature tests, depending on the case. In case of bugfixes a test, that fails in the appropriate -case should be implemented, to make regression tests possible for further changes in the application. For the backend the -api can be tested by using feature tests and other functions just with unit tests. For more information about tests -checkout the [Laravel testing guides](https://laravel.com/docs/11.x/testing). +case should be implemented, to make regression tests possible for further changes in the application. -To run the integration tests against a real BigBlueButton server you need to set the test server in -the `.env` file: - -```bash -BBB_TEST_SERVER_HOST=https://your-bbb-server.com/bigbluebutton/ -BBB_TEST_SERVER_SECRET=your-secret -``` - -### Frontend - -Since the refactoring of the frontend to Vue 3 and PrimeVue, the frontend tests are still a work in progress. -We have started adding some basic [Cypress](https://www.cypress.io) tests, but they are currently incomplete, and code coverage is not yet implemented. -The old Vitest tests have been removed. If you need to reference them, please check the old branches. - -If you have experience with Cypress and would like to contribute, -your help is welcome! Please feel free to create a PR. +Please have a look into the [Testing](./07-testing.md) documentation to learn more about the testing strategies +and how to run the tests. ## Submit changes diff --git a/docs/docs/development/07-testing.md b/docs/docs/development/07-testing.md index 164dd9581..7d1ca2be3 100644 --- a/docs/docs/development/07-testing.md +++ b/docs/docs/development/07-testing.md @@ -4,11 +4,11 @@ title: Testing Our goal is to build a high-quality product by focusing equally on robust architecture, smooth UI/UX design, and accessible experiences. By combining these elements, we aim to deliver a stable, reliable, and user-friendly platform. -Maintaining product quality is an ongoing process that requires consistent effort. To prevent quality degradation over time, we have established a comprehensive testing strategy, which includes the following components: +Maintaining product quality is an ongoing process that requires consistent effort. To prevent quality degradation over time, we have established a comprehensive testing strategy that includes the following components: 1. **PHP Backend Testing** - - We employ multiple levels of backend tests, including unit tests, integration tests, and more. These tests ensure that each component functions correctly and integrates seamlessly with others. + - We use several levels of backend testing, including unit testing, integration testing and more. These tests ensure that each component works correctly and integrates seamlessly with others. 2. **Frontend Testing** @@ -16,17 +16,17 @@ Maintaining product quality is an ongoing process that requires consistent effor 3. **System Testing** - - We conduct system tests to verify that the entire application operates as expected. These include ensuring the Docker container initializes correctly and confirming that the frontend and backend work together seamlessly. + - We perform system testing to verify that the entire application works as expected. This includes ensuring that the Docker container initialises correctly, and confirming that the frontend and backend work together seamlessly. 4. **Visual Regression Testing** - - Visual regression tests are our latest addition. These tests help us quickly identify and address visual inconsistencies, ensuring a consistent user interface and experience. + - Visual regression testing is our latest addition. These tests help us to quickly identify and fix visual inconsistencies, ensuring a consistent user interface and experience. -By combining these testing methodologies, we ensure that PILOS maintains its high standards of quality and continues to meet user expectations. +By combining these testing methods, we ensure that PILOS maintains its high quality standards and continues to meet user expectations. ## PHP Backend Testing with PHPUnit The backend of PILOS is written in PHP and uses the Laravel framework. -Laravel provides a robust PHPUnit based test system that allows us to test the application at different levels. +Laravel provides a robust PHPUnit based testing system that allows us to test the application on different levels. You can learn more about Laravel testing in the [official documentation](https://laravel.com/docs/11.x/testing). To run the full test suite using a real BigBlueButton server you need to set the test server in the `.env` file: @@ -65,30 +65,37 @@ For testing Cypress is used. You can learn more about Cypress in the [official d The tests stub the backend API requests with fixtures. This allows us to test the frontend in isolation and without the need for a running backend. It also helps to simulate a lot of different edge cases and error scenarios. -To run the frontend tests, you first have to install cypress outside the container: +### Installing Cypress + +First you need to fulfill the [system requirements](https://docs.cypress.io/app/get-started/install-cypress#System-requirements) of Cypress. +Please make sure your OS is supported and all required packages are installed. + +If the system requirements are met, you can install Cypress on your local machine: ```bash npx cypress install ``` -Next you need to have the frontend either running compiled (`./sail npm run build`) or in development mode (`./sail npm run dev`). +### Running Cypress Tests + +First you need to have the frontend either running compiled (`./sail npm run build`) or in development mode (`./sail npm run dev`). You can then run the tests in the interactive mode with: ```bash -npx cypress open +npm run cypress:open ``` Or you can run the tests headless with: ```bash -npx cypress run +npm run cypress:run ``` ### Coverage -To generate a code coverage report the frontend first has to be instrumented to the coverage can be measured. -This can build the frontend with the coverage instrumentation: +To generate a code coverage report, the frontend must first be instrumented so that the coverage can be measured. +This command can be used to build the frontend with coverage instrumentation: ```bash ./sail npm run build -- --config vite.config.coverage.js @@ -97,7 +104,7 @@ This can build the frontend with the coverage instrumentation: After that you can run the tests with the coverage report: ```bash -npx cypress run +npm run cypress:run ``` To generate a code coverage report in HTML format you can use the following command: @@ -109,42 +116,46 @@ To generate a code coverage report in HTML format you can use the following comm ## System Testing with Cypress System tests are used to verify that the entire application operates as expected. +To ensure that the frontend and backend work together seamlessly, the tests are using the real backend API and a real BigBlueButton server. +The tests are also using Cypress for functional testing. -To run the system tests we have a separate docker-compose file that needs to be started: +To run the system tests a separate docker-compose file needs to be started: ```bash docker compose -f compose.test.yml -f compose.test.local.yml up -d ``` +Before running the tests make sure you have correctly installed cypress as described in the [Installing cypress](#installing-cypress) section. + You can then run the tests in the interactive mode with: ```bash -npx cypress open -P ./tests/System +npm run cypress:open -- -P ./tests/System ``` Or you can run the tests headless with: ```bash -npx cypress run -P ./tests/System +npm run cypress:run -- -P ./tests/System ``` ## Visual Regression Testing with Happo.io -We aim to cover as much functionality as possible with our frontend tests. However, testing every aspect of the application can be challenging -and sometimes simply not possible, e.g. layout shifts, svg rendered charts, etc. +We aim to cover as much functionality as possible with our frontend tests. However, testing every aspect of the application can be challenging and sometimes simply not possible, e.g. layout shifts, svg rendered charts, etc. -PILOS relies on several third-party libraries, such as UI Components (PrimeVue), Charts (Chart.js), and others. While these libraries are generally high quality and well-tested, they may occasionally introduce bugs, regressions, or integration issues within our application. Frequent updates to these libraries further necessitate ensuring that our application continues to look and behave as expected. +PILOS relies on several third-party libraries, such as UI components (PrimeVue), charts (Chart.js) and others. While these libraries are generally high quality and well tested, they may occasionally introduce bugs, regressions or integration issues into our application. Frequent updates to these libraries also require us to ensure that our application continues to look and behave as expected. -Identifying visual regressions manually is a time-consuming and error-prone process, which has been a significant pain point in the past. To address this, we have implemented visual regression testing using [Happo](https://happo.io/). +Manually identifying visual regressions is a time-consuming and error-prone process that has been a significant pain point in the past. To address this, we implemented visual regression testing using [Happo](https://happo.io/). -Happo is a powerful tool that enables us to take screenshots of our application and compare them against previous versions to detect visual changes. -Integrating Happo into our existing Cypress test system was straightforward, and we will extend its use to all areas of the application where visual changes are expected. +Happo is a powerful tool that allows us to take screenshots of our application and compare them to previous versions to detect visual changes. Integrating Happo into our existing Cypress test system was straightforward and we will extend its use to all areas of the application where visual changes are expected. + +By using Happo for visual regression testing, we can maintain a high standard of visual quality in our application while minimising manual effort. ### Automated Testing Environment To ensure consistent results, we run visual regression tests exclusively in the GitHub Action runner. This controlled environment minimizes discrepancies and provides reliable feedback. -When a screenshot differs from the baseline, we can review the changes and determine whether it indicate a bug or an intentional update. +If a screenshot differs from the baseline, we can review the changes and determine whether this is a bug or an intentional change. ### Adding Screenshots in Tests @@ -158,10 +169,10 @@ cy.get("button").happoScreenshot(); Although visual regression tests run exclusively in the GitHub Action runner, you can write and debug tests locally. Use the following command to open Cypress for interactive debugging: +Before running the tests make sure you have correctly installed cypress as described in the [Installing cypress](#installing-cypress) section. + ```bash -npx cypress open -P ./tests/Visual --config experimentalInteractiveRunEvents=true +npm run cypress:open -- -P ./tests/Visual --config experimentalInteractiveRunEvents=true ``` This setup allows you to ensure your tests are functioning correctly before committing changes. - -By leveraging Happo for visual regression testing, we can maintain a high standard of visual quality in our application while minimizing manual effort.