Table of Contents

Development

Pick an environment

There are two development environments, and you need to install at least one of them first.

  • The Docker containerized environment is the preferred development environment. The Docker images are already provisioned, so setup is faster. It is a better model for the production environment, and after the planned rehost will be almost exactly the same. It does not yet have all of the features of the Vagrant environment, and it is currently slower for many development tasks.
  • The Vagrant-managed VM is the mature but deprecated development environment. It can be tricky to provision. It is often a poor model for the production environment, and can not be used to test infrastructure changes.

Docker and Vagrant can be used at the same time on the same “host machine” (your laptop or desktop computer).

Basic Docker usage

Edit files as usual on your host machine; the current directory is mounted via Docker host mounting at /app within the kuma_web_1 and other containers. Useful docker sub-commands:

docker exec -it kuma_web_1 bash  # Start an interactive shell
docker logs kuma_web_1           # View logs from the web container
docker-compose logs -f           # Continuously view logs from all containers
docker restart kuma_web_1        # Force a container to reload
docker-compose stop              # Shutdown the containers
docker-compose up -d             # Start the containers
docker-compose rm                # Destroy the containers

There are make shortcuts on the host for frequent commands, such as:

make up         # docker-compose up -d
make bash       # docker exec -it kuma_web_1 bash
make shell_plus # docker exec -it kuma_web_1 ./manage.py shell_plus

Run all commands in this doc in the kuma_web_1 container after make bash.

Basic Vagrant usage

Edit files as usual on your host machine; the current directory is mounted via NFS at /home/vagrant/src within the VM. Updates should be reflected without any action on your part. Useful vagrant sub-commands:

vagrant ssh     # Connect to the VM via ssh
vagrant suspend # Sleep the VM, saving state
vagrant halt    # Shutdown the VM
vagrant up      # Boot up the VM
vagrant destroy # Destroy the VM

Run all commands in this doc on the VM after vagrant ssh.

Running Kuma

When the Docker container environment is started (make up or similar), all of the services are also started. The development instance is available at http://localhost:8000.

The Vagrant environment runs everything in a single VM. It runs MySQL, ElasticSearch, Apache, and other “backend” services whenever the VM is running. There are additional Kuma-specific services that are configured in Procfile, and are run with:

foreman start

The Vagrant development instance is then available at https://developer-local.allizom.org.

Running the tests

One way to confirm that everything is working, or to pinpoint what is broken, is to run the test suite.

Django tests

Run the Django test suite:

make test

For more information, see the test documentation.

Front-end tests

To run the front-end (selenium) tests, see Client-side Testing.

Kumascript tests

If you’re changing Kumascript, be sure to run its tests too. See https://github.com/mozilla/kumascript.

Front-end Development and Compiling Sass files

Sass files need to be compiled for changes to take effect, but don’t worry, with DEBUG=True (which is the default for local development), the compilation is done automatically by Gulp.

When doing front-end development on your local machine, you’ll need to run the following in its own shell from the root directory of your local Kuma repository:

gulp

This will watch for changes to all files under ./kuma/static. If the file is a Sass file (.scss or .sass), it will trigger a recompile of all top-level .scss files, which will then be copied over to ./static (and immediately available to your local server). If the file is not a Sass file, it will be copied to ./static with no compile step.

It is currently faster for local development to compile Sass using gulp-sass instead of Django Pipeline. This may change in the future.

If you haven’t already installed Node.js and gulp on your local machine, see Prepare for Front-end Development.

By default DEBUG=True in docker-compose.yml, and in that mode, as mentioned above, source files are compiled on-demand. If for some reason you want to run with DEBUG = False, just remember that source files will no longer be compiled on-demand. Instead, after every change to one or more source files, you’ll have to do the following:

docker-compose exec web ./manage.py collectstatic
docker-compose restart web

in order for your changes to be visible.

Database migrations

Apps are migrated using Django’s migration system. To run the migrations:

manage.py migrate

If your changes include schema modifications, see the Django documentation for the migration workflow.

Coding conventions

See CONTRIBUTING.md for details of the coding style on Kuma.

New code is expected to have test coverage. See the Test Suite docs for tips on writing tests.

Managing dependencies

Python dependencies

Kuma tracks its Python dependencies with pip. See the README in the requirements folder for details.

Front-end dependencies

Front-end dependencies are managed by Bower and checked into the repository. Follow these steps to add or upgrade a dependency:

  1. On the host, update bower.json.
  2. (Docker only) In the container, install git (apt-get install -y git).
  3. (Docker only) In the container, install bower-installer (npm install -g bower-installer).
  4. In the VM or container, install the dependency (bower-installer).
  5. On the host, prepare the dependency to be committed (git add path/to/dependency).

Front-end dependencies that are not already managed by Bower should begin using this approach the next time they’re upgraded.

Advanced configuration

Environment variables are used to change the way different components work. There are a few ways to change an environment variables:

  • Exporting in the shell, such as:

    export DEBUG=True;
    ./manage.py runserver
    
  • A one-time override, such as:

    DEBUG=True ./manage.py runserver
    
  • Changing the environment list in docker-compose.yml.

  • Creating a .env file in the repository root directory.

One variable you may wish to alter for local development is DEBUG_TOOLBAR, which, when set to True, will enable the Django Debug Toolbar:

DEBUG_TOOLBAR=True

Note that enabling the Debug Toolbar can severely impact response time, adding around 4 seconds to page load time.

The Docker environment

Running docker-compose will create and run several containers, and each container’s environment and settings are configured in docker-compose.yml. The settings are “baked” into the containers created by docker-compose up.

To override a container’s settings for development, use a local override file. For example, the web service runs in container kuma_web_1 with the default command “gunicorn -w 4 --bind 0.0.0.0:8000 --timeout=120 kuma.wsgi:application”. A useful alternative for debugging is to run a single-threaded process that loads the Werkzeug debugger on exceptions (see docs for runserver_plus), and that allows for stepping through the code with a debugger. To use this alternative, create an override file docker-compose.dev.yml:

version: "2"
services:
  web:
    command: ./manage.py runserver_plus 0.0.0.0:8000
    stdin_open: true
    tty: true

This is similar to “docker run -it <image> ./manage.py runserver_plus”, using all the other configuration items in docker-compose.yml. Apply the custom setting with:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d

You can then add pdb breakpoints to the code (import pdb; pdb.set_trace) and connect to the debugger with:

docker attach kuma_web_1

To always include the override compose file, add it to your .env file:

COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

A similar method can be used to override environment variables in containers, run additional services, or make other changes. See the docker-compose documentation for more ideas on customizing the Docker environment.

The Vagrant environment

It is easiest to configure Vagrant with a .env file, so that overrides are used when vagrant up is called. A sample .env could contain:

VAGRANT_MEMORY_SIZE=4096
VAGRANT_CPU_CORES=4
# Comments are OK, for documentation and to disable settings
# VAGRANT_ANSIBLE_VERBOSE=true

Configuration variables that are available for Vagrant:

  • VAGRANT_NFS

    Default: true (Windows: false) Whether or not to use NFS for the synced folder.

  • VAGRANT_MEMORY_SIZE

    The size of the Virtualbox VM memory in MB. Default: 2048.

  • VAGRANT_CPU_CORES

    The number of virtual CPU core the Virtualbox VM should have. Default: 2.

  • VAGRANT_IP

    The static IP the Virtualbox VM should be assigned to. Default: 192.168.10.55.

  • VAGRANT_GUI

    Whether the Virtualbox VM should boot with a GUI. Default: false.

  • VAGRANT_ANSIBLE_VERBOSE

    Whether the Ansible provisioner should print verbose output. Default: false.

  • VAGRANT_CACHIER

    Whether to use the vagrant-cachier plugin to cache system packages between installs. Default: true.

The database

The database connection is defined by the environment variable DATABASE_URL, with these defaults:

DATABASE_URL=mysql://kuma:kuma@localhost:3306/kuma              # Vagrant
DATABASE_URL=mysql://root:kuma@mysql:3306/developer_mozilla_org # Docker

The format is defined by the dj-database-url project:

DATABASE_URL=mysql://user:password@host:port/database

If you configure a new database, override DATABASE_URL to connect to it. To add an empty schema to a freshly created database:

./manage.py migrate

To connect to the database specified in DATABASE_URL, use:

./manage.py dbshell

Asset generation

Kuma will automatically run in debug mode, with the DEBUG setting turned to True. Setting DEBUG=False will put you in production mode and generate/use minified (compressed) and versioned (hashed) assets. To emulate production, and test compressed and hashed assets locally:

  1. Set the environment variable DEBUG=false.
  2. Start (docker-compose up -d) or restart (``docker-compose restart`) your Docker services.
  3. Run docker-compose exec web make build-static.
  4. Restart the web process using docker-compose restart web.

Secure cookies

To prevent error messages like “Forbidden (CSRF cookie not set.):”, set the environment variable:

CSRF_COOKIE_SECURE = false

This is the default in Docker, which does not support local development with HTTPS.

Deis Workflow Demo instances

You can deploy a hosted demo instance of Kuma by following these steps:

  1. Create a new branch, you cannot create a demo from the master branch.

  2. from the Kuma project root directory, run the following command:

    make create-demo
    
  3. Your demo will be accessible within about 10 minutes at:

    https://mdn-demo-<your_branch_name>.virginia.moz.works
    
  4. Mozilla SRE’s will periodically remove old instances

  5. Connecting to the demo database instance

If you have access to Kubernetes, you can run the following command to connect to the MySQL instance:

MY_GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
DEMO_MYSQL_POD=$(kubectl -n "mdn-demo-${MY_GIT_BRANCH}" get pods | grep "^mysql" | awk '{ print $1 }')
kubectl -n "mdn-demo-${MY_GIT_BRANCH}" exec -it ${DEMO_MYSQL_POD} bash

mysql -p developer_mozilla_org

Note: if you copy and paste the code above into a bash terminal and are wondering why the commands don’t appear in your bash history, it’s because there’s whitespace at the beginning of the line.

Maintenance Mode

Maintenance mode is a special configuration for running Kuma in read-only mode, where all operations that would write to the database are blocked. As the name suggests, it’s intended for those times when we’d like to continue to serve documents from a read-only copy of the database, while performing maintenance on the master database.

For local Docker-based development in maintenance mode:

  1. If you haven’t already, create a read-only user for your local MySQL database:

    docker-compose up -d
    docker-compose exec web mysql -h mysql -u root -p
    (when prompted for the password, enter "kuma")
    mysql> source ./scripts/create_read_only_user.sql
    mysql> quit
    
  2. Create a .env file in the repository root directory, and add these settings:

    MAINTENANCE_MODE=True
    DATABASE_USER=kuma_ro
    

    Using a read-only database user is not required in maintenance mode. You can run in maintenance mode just fine with only this setting:

    MAINTENANCE_MODE=True
    

    and going with a database user that has write privileges. The read-only database user simply provides a level of safety as well as notification (for example, an exception will be raised if an attempt to write the database slips through).

  3. Update your local Docker instance:

    docker-compose up -d
    
  4. You may need to recompile your static assets and then restart:

    docker-compose exec web make build-static
    docker-compose restart web
    

You should be good to go!

There is a set of integration tests for maintenance mode. If you’d like to run them against your local Docker instance, you must first have done the following:

  1. Loaded the latest sample database (see Load the Sample Database).

  2. Ensured that the test document “en-US/docs/User:anonymous:uitest” has been rendered (all of its macros have been executed). You can check this by browsing to http://localhost:8000/en-US/docs/User:anonymous:uitest. If there is no message about un-rendered content, you are good to go. If there is a message about un-rendered content, you will have to put your local Docker instance back into non-maintenance mode, and render the document:

    and then put your local Docker instance back in maintenance mode:

    • Configure your .env file for maintenance mode:

      MAINTENANCE_MODE=True
      DATABASE_USER=kuma_ro
      
    • docker-compose up -d

  3. Configure your environment with DEBUG=False because the maintenance-mode integration tests check for the non-debug version of the not-found page:

    DEBUG=False
    MAINTENANCE_MODE=True
    DATABASE_USER=kuma_ro
    

    This, in turn, will also require you to recompile your static assets:

    docker-compose up -d
    docker-compose exec web ./manage.py compilejsi18n
    docker-compose exec web ./manage.py collectstatic
    docker-compose restart web
    

Now you should be ready for a successful test run:

py.test -m "maintenance_mode and not search" tests/functional --base-url http://localhost:8000 --driver Chrome --driver-path /path/to/chromedriver

Note that the “search” tests are excluded. This is because the tests marked “search” are not currently designed to run against the sample database.