forgejo/tests/e2e/README.md

End to end tests

Thank you for your effort to provide good software tests for Forgejo. Please also read the general testing instructions in the Forgejo contributor documentation and make sure to also check the Playwright documentation for further information.

This file is meant to provide specific information for the integration tests as well as some tips and tricks you should know.

Feel free to extend this file with more instructions if you feel like you have something to share!

How to run the tests?

Before running any tests, please ensure you perform a clean frontend build:

make clean frontend

Whenever you modify frontend code (i.e. JavaScript and CSS files), you need to create a new frontend build.

For tests that require interactive Git repos, you also need to ensure a Forgejo binary is ready to be used by Git hooks. For this, you additionally need to run

make TAGS="sqlite sqlite_unlock_notify" backend

Install dependencies

Browsertesting is performed by playwright. You need certain system libraries and playwright will download required browsers. Playwright takes care of this when you run:

npx playwright install-deps

Note

On some operating systems, the installation of missing libraries can complicate testing certain browsers. It is often not necessary to test with all browsers locally. Choosing either Firefox or Chromium is fine.

Run all tests

If you want to run the full test suite, you can use

make test-e2e-sqlite

Interactive testing

We recommend that you use interactive testing for the development. After you performed the required builds, you should use one shell to start the debugserver (and leave it running):

make test-e2e-debugserver

It allows you to explore the test data in your local browser, and playwright to perform tests on it.

Note

The modifications persist while the debugserver is running. If you modified things, it might be useful to restart it to get back to a fresh state. While writing playwright tests, you either need to ensure they are resilient against repeated runs (e.g. when only creating new content), or that they restore the initial state for the next browser run.

With the playwright UI:

Playwright ships with an integrated UI mode which allows you to run individual tests and to debug them by seeing detailed traces of what playwright does. Launch it with:

npx playwright test --ui

Running individual tests

npx playwright test actions.test.e2e.js:9

First, specify the complete test filename, and after the colon you can put the linenumber where the test is defined.

With VSCodium or VSCode

To debug a test, you can also use "Playwright Test" for VScodium or VSCode.

Run all tests via local act_runner

If you have a forgejo runner, you can use it to run the test jobs:

forgejo-runner exec -W .forgejo/workflows/testing.yml -j test-e2e

Note that the CI workflow has some logic to run tests based on changed files only. This might conflict with your local setup and not run all the desired tests because it might only look at file changes in your latest commit.

Run e2e tests with another database

This approach is not currently used, neither in the CI/CD nor by core contributors on their lcoal machines. It is still documented for the sake of completeness: You can also perform e2e tests using MariaDB/MySQL or PostgreSQL if you want.

Setup a MySQL database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql

Running individual tests

Example command to run example.test.e2e.js test file:

Note

Unlike integration tests, this filtering is at the file level, not function

For SQLite:

make test-e2e-sqlite#example

Visual testing

Warning

This is not currently used by most Forgejo contributors. Your help to improve the situation and allow for visual testing is appreciated.

Although the main goal of e2e is assertion testing, we have added a framework for visual regress testing. If you are working on front-end features, please use the following:

• Check out main, make clean frontend, and run e2e tests with VISUAL_TEST=1 to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert it passes.
• Check out your branch, make clean frontend, and run e2e tests with VISUAL_TEST=1. You should be able to assert you front-end changes don't break any other tests unintentionally.

VISUAL_TEST=1 will create screenshots in tests/e2e/test-snapshots. The test will fail the first time this is enabled (until we get visual test image persistence figured out), because it will be testing against an empty screenshot folder.

ACCEPT_VISUAL=1 will overwrite the snapshot images with new images.

Tips and tricks

If you know noteworthy tests that can act as an inspiration for new tests, please add some details here.

Run tests very selectively

Browser testing can take some time. If you want to iterate fast, save your time and only run very selected tests. Use only one browser.

Skip Safari if it doesn't work

Many contributors have issues getting Safari (webkit) and especially Safari Mobile to work.

At the top of your test function, you can use:

test.skip(workerInfo.project.name === 'Mobile Safari', 'Unable to get tests working on Safari Mobile.');

Don't forget the formatting.

When writing tests without modifying other frontend code, it is easy to forget that the JavaScript test files also need formatting.

Run make lint-frontend-fix.

Define new repos

Take a look at declare_repos_test.go to see how to add your repositories. Feel free to improve the logic used there if you need more advanced functionality (it is a simplified version of the code used in the integration tests).

Accessibility testing

If you can, perform automated accessibility testing using AxeCore.

Take a look at shared/forms.js and some other places for inspiration.

List related files coverage

To speed up the CI pipelines and avoid running expensive tests too often, only a selection of tests is run by default, based on the changed files.

At the top of each playwright test file, list the files or file patterns that are covered by your test. Often, these are files that you modified for your feature or bugfix, or that you looked at (and might still have open in your IDE), because your fix depends on their behaviour.

Which files to watch?

The set of files your test "watches" depends on the kind of test you write. If you only test for the presence of an element and do no accessibility or placement checks, you won't detect broken visual appearance and there is little reason to watch CSS files.

However, if your test also checks that an element is correctly positioned (e.g. that it does not overflow the page), or has accessibiltiy properties (includes colour contrast), also list stylesheets that define the behaviour your test depends on.

Watching the place that generate the selectors you use (typically templates, but can also be JavaScript) is a must, to ensure that someone modifying the markup notices that your selectors fail (e.g. because an id or class was renamed).

If you are unsure about the exact set of files, feel free to ask other contributors.

How to specify the patterns?

You put filenames and patterns as blocks between two // @watch comments. An example that watches changes on (in order) a single file, a full recursive subfolder, two files with a shorthand pattern, and a set of files with a certain ending:

// @watch start
// templates/webhook/shared-settings.tmpl
// templates/repo/settings/**
// web_src/css/{form,repo}.css
// web_src/css/modules/*.css
// @watch end

The patterns are evaluated on a "first-match" basis. Under the hood, gobwas/glob is used.