The tests directory comprises of:
/functionalcontains pytest tests.
/pagescontains Python pypom page objects.
/utilscontains helper functions.
Setting up test virtual environment¶
In the terminal go to the directory where you have downloaded kuma.
Create a virtual environment for the tests to run in (the example uses the name mdntests):
$ virtualenv mdntests
Activate the virtual environment:
$ source mdntests/bin/activate
Make sure the version of pip in the virtual environment is high enough to support hashes:
$ pip install "pip>=8"
Install the requirements the tests need to run:
$ pip install -r requirements/test.txt
You may want to add your virtual environment folder to your local .gitignore file.
Running the tests¶
Before running the tests you will need to download the driver for the browser you want to test in. Drivers are listed and linked to in the pytest-selenium documentation.
The minimum amount of information necessary to run the tests against the staging
server is the directory the tests are in, which driver to use, and the
subset of tests to run (by specifying a marker or marker expression like
-m "not login", for more information see Markers). Currently, all of the
tests except the tests marked “login” should run successfully against the staging
In the virtual environment run:
$ py.test -m "not login and not maintenance_mode" tests/functional/ --driver Chrome --driver-path /path/to/chromedriver
You will be prompted “Do you want the application ‘python’ to accept incoming network connections?” The tests seem to run fine no matter how you answer.
This basic command can be modified to use different browsers, to run the tests against a local environment, or to run tests concurrently.
Only running tests in one file¶
Add the name of the file to the test location:
$ py.test -m "not login and not maintenance_mode" tests/functional/test_search.py --driver Chrome --driver-path /path/to/chromedriver
Run the tests against a different url¶
By default the tests will run against the staging server. If you’d like to run
the tests against a different URL (e.g., your local environment) pass the
desired URL to the command with
$ py.test -m "not login and not maintenance_mode" tests/functional/ --base-url http://localhost:8000 --driver Chrome --driver-path /path/to/chromedriver
Run the tests in parallel¶
By default the tests will run one after the other but you can run several at
the same time by specifying a value for
$ py.test -m "not login and not maintenance_mode" tests/functional/ -n auto --driver Chrome --driver-path /path/to/chromedriver
Run tests on SauceLabs¶
Running the tests on SauceLabs will allow you to test browsers not on your host machine.
Log in and obtain your Remote Access Key from user settings.
Run a test specifying
SauceLabsas your driver, and pass your credentials and the browser to test:
$ SAUCELABS_USERNAME=thedude SAUCELABS_API_KEY=123456789 py.test -m "not login and not maintenance_mode" tests/functional/ --driver SauceLabs --capability browsername MicrosoftEdge
Alternatively you can save your credentials in a configuration file so you don’t have to type them each time.
Run tests on MDN’s Continuous Integration (CI) infrastructure¶
If you have commit rights on the mozilla/kuma GitHub repository you can run the UI tests using the MDN CI Infrastructure. Just force push to mozilla/kuma@stage-integration-tests to run the tests against https://developer.allizom.org.
You can check the status, progress, and logs of the test runs at MDN’s Jenkins-based multi-branch pipeline.
MDN CI Infrastructure¶
The MDN CI infrastructure is a Jenkins-based, multi-branch pipeline. The pipelines for all branches are defined by the Jenkinsfile and the files under the Jenkinsfiles directory. The basic idea is that every branch may have its own custom pipeline steps and configuration.
Jenkins will auto-discover the steps and configuration by checking within the
Jenkinsfiles directory for a Groovy (
.groovy) and/or YAML (
file with the same name as the branch. For example, the
“stage-integration-tests” branch has a
Jenkinsfiles/stage-integration-tests.yml file which will be
loaded as configuration and used to determine what to do next (load and
run the Groovy script specified by its
pipeline.script setting -
Jenkinsfiles/integration-tests.groovy - and the script, in turn, will use
the dictionary of values provided by the
job setting defined within the
Note that the YAML files for the integration-test branches provide settings for configuring things like the version of Selenium to use, the number of Selenium nodes to spin-up, the Dockerfile to use to build the container, the URL to test against, and which subset of integration tests to run (via a pytest marker expression, see Markers).
The integration-test Groovy files use a number of global Jenkins functions that were developed to make the building, running and pushing of Docker containers seamless (as well as other cool stuff, see mozmar/jenkins-pipeline). They allow us to better handle situations that have been painful in the past, like the stopping of background Docker containers.
The “prod-integration-tests” branch also has its own
Jenkinsfiles/prod-integration-tests.yml file. It’s identical to the YAML
file for the “stage-integration-tests” branch except that it specifies that
the tests should be run against the production server rather than the staging
server (via its
Similarly, the “master” branch has it’s own pipeline, but instead of being configured by a YAML file, the entire pipeline is defined within its Jenkinsfiles/master.groovy file.
The pipeline for any other branch which does not provide its own Groovy and/or YAML file will follow that defined by the Jenkinsfiles/default.groovy file.
You can check the status, progress, and logs of any pipeline runs via MDN’s Jenkins-based multi-branch pipeline.
Tests are considered destructive unless otherwise indicated. Tests that create, modify, or delete data are considered destructive and should not be run in production.
These tests should be the critical baseline functional tests.
New instances of kuma have empty databases so only a subset of tests can be run against them. These tests are marked with
These tests require the testing accounts to exist on the target site. For security reasons these accounts will not be on production. Exclude these tests with
-m "not login"
These tests can be run against an instance of Kuma in maintenance mode. They cover all of the key aspects, such as testing all of the endpoints that should redirect to the maintenance-mode description page, as well as ensuring that the buttons and links that initiate write operations (like editing a document) are hidden.