Client-side testing¶

Kuma has a suite of functional tests using pytest. All these test suites live in the /tests/ directory.

The tests directory comprises of:

/headless contains tests that don’t require a browser
/utils contains helper functions.

Setting up test environment¶

In the terminal go to the directory where you have downloaded kuma.
Install the requirements the tests need to run:
```
$ poetry install
```

Running the tests¶

The minimum amount of information necessary to run the tests against the staging server is the directory the tests are in. Currently, all of the tests should run successfully against the staging server.:

$ poetry run pytest tests

This basic command can be modified to run the tests against a local environment, or to run tests concurrently.

Only running tests in one file¶

Add the name of the file to the test location:

$ poetry run pytest tests/headless/test_cdn.py

Run the tests against a different url¶

By default the tests will run against the staging server. If you’d like to run the tests against a different URL (e.g., your local environment) pass the desired URL to the command with --base-url:

$ poetry run pytest tests --base-url http://mdn.localhost:8000

Run the tests in parallel¶

By default the tests will run one after the other but you can run several at the same time by specifying a value for -n:

$ poetry run pytest tests -n auto

Using Alternate Environments¶

Run tests on MDN’s Continuous Integration (CI) infrastructure¶

If you have commit rights on the mdn/kuma GitHub repository you can run the functional tests using the MDN CI Infrastructure. Just force push to mdn/kuma@stage-integration-tests to run the tests against https://developer.allizom.org.

You can check the status, progress, and logs of the test runs at MDN’s Jenkins-based multi-branch pipeline.

MDN CI Infrastructure¶

The MDN CI infrastructure is a Jenkins-based, multi-branch pipeline. The pipelines for all branches are defined by the Jenkinsfile and the files under the Jenkinsfiles directory. The basic idea is that every branch may have its own custom pipeline steps and configuration.

Jenkins will auto-discover the steps and configuration by checking within the Jenkinsfiles directory for a Groovy (.groovy) and/or YAML (.yml) file with the same name as the branch. For example, the “stage-integration-tests” branch has a Jenkinsfiles/stage-integration-tests.yml file which will be loaded as configuration and used to determine what to do next (load and run the Groovy script specified by its pipeline.script setting - Jenkinsfiles/integration-tests.groovy - and the script, in turn, will use the dictionary of values provided by the job setting defined within the configuration).

Note that the YAML files for the integration-test branches provide settings for configuring things like the Dockerfile to use to build the container, and the URL to test against.

The integration-test Groovy files use a number of global Jenkins functions that were developed to make the building, running and pushing of Docker containers seamless (as well as other cool stuff, see mozmar/jenkins-pipeline). They allow us to better handle situations that have been painful in the past, like the stopping of background Docker containers.

The “prod-integration-tests” branch also has its own Jenkinsfiles/prod-integration-tests.yml file. It’s identical to the YAML file for the “stage-integration-tests” branch except that it specifies that the tests should be run against the production server rather than the staging server (via its job.base_url setting).

Similarly, the “master” branch has it’s own pipeline, but instead of being configured by a YAML file, the entire pipeline is defined within its Jenkinsfiles/master.groovy file.

The pipeline for any other branch which does not provide its own Groovy and/or YAML file will follow that defined by the Jenkinsfiles/default.groovy file.

You can check the status, progress, and logs of any pipeline runs via MDN’s Jenkins-based multi-branch pipeline.

Markers¶

nondestructive

Tests are considered destructive unless otherwise indicated. Tests that create, modify, or delete data are considered destructive and should not be run in production.
smoke

These tests should be the critical baseline functional tests.

Guidelines for writing tests¶

See Bedrock and the Web QA Style Guide.