Deploying MDN

MDN is served from Amazon Web Services in the US West (Oregon) datacenters. MDN is deployed as Docker containers managed by Kubernetes. The infrastructure is defined within MDN’s infra repository, including the tools used to deploy and maintain MDN.

Deploying new code to AWS takes several steps, and about 60 minutes for the full process. The deployer is responsible for emergency issues for the next 24 hours. There are 1-3 deployments per week, usually between 7 AM and 1 PM Pacific, Monday through Thursday.

Note

This page describes deployment performed by MDN staff. It requires additional setup and permissions not described here. Deployments will not work for non-staff developers, and should not be attempted.

Pre-Deployment

Before deploying, a staff member should:

  • (Once a week) Check that the latest mdn-browser-compat-data node package is included in the KumaScript image. The latest package is installed when the image is created, during the RUN npm update step, and can be verified by reading the build logs or running the image.

  • (Once a week) Update the locale and kumascript submodules. The locale submodule must be updated to apply new translations to production. The kumascript submodule is not used for deployment, but updating it keeps the development environment in sync.

    • Commit or stash any changes in your working copy.

    • Create a pre-push branch from master:

      git fetch origin
git checkout -b pre-push-`date +"%Y-%m-%d"` origin/master

    • Update the submodules:

      git submodule update --remote
git commit -m "Updating submodules" kumascript locale

    • Push new localizable strings. This step is only necessary when there are new strings. The TravisCI job TOXENV=locales checks for new strings, and you can examine the output of the recent master build to see if there are differences. At the same time, it’s not too hard to run locally and then revert if there are no useful changes.

      1. On the host system, checkout the master locale branch:

        cd locale
git checkout master
git pull

      2. Update kuma/settings/common.py, and bump the version in PUENTE['VERSION'].

      3. Inside the development environment, extract and rebuild the translations:

        make localerefresh

      4. On the host system (still in the locale subfolder), review the changes to source English strings:

        git diff templates/LC_MESSAGES

      5. If there are useful changes (ignore comment lines and look for msgid as the changed line), commit the files in the locale submodule:

        git add --all .
git commit

        For the commit message, use the PUENTE['VERSION'] in the commit subject, and summarize the string changes in the commit body, like:

        Update strings 2018.08

* Add "View All" text for document history pagination

        Attempt to push to the mdn-l10n repository:

        git push

        If this fails, do not force with –force, or attempt to pull and create a merge commit. Someone has added a translation while you were working, and you need to start over to preserve their work:

        git fetch
git reset --hard @{u}

        This resets your locale submodule to the new master. Start over on step 3 (make localerefresh).

        If the push to mdn-l10n is a success, commit your Kuma changes:

        cd ..
git commit kuma/settings/common.py locale

        You can use the same commit message used for the locale commit.

      6. If there are no new strings, throw the changes away, starting in the locale folder:

        git checkout -- .
cd ..
git checkout -- kuma/settings/common.py

    • Push the branch and open a pull request:

      git push -u origin pre-push-`date +"%Y-%m-%d"`

    • Check that tests pass. The TOXENV=locales job, which uses Dennis to check strings, is the job that occasionally fails. This is due to a string change committed in Pontoon that will break rendering in production. If this happens, fix the incorrect translation, push to master on mdn-l10n, and update the submodule in the pull request. Wait for the tests to pass.

    • Merge the pull request. A “Rebase and merge” is preferred, to avoid a merge commit with little long-term value. Delete the pre-push branch.

  • Check the latest images from master are built by Jenkins (Kuma master build, KumaScript master build, both private to staff), and uploaded to the quay.io (pronounced “Key dot I O”) repositories (Kuma images, KumaScript images).

Deploy to Staging

The staging site is located at https://developer.allizom.org. It runs on the same Kuma code as production, but against a different database, other backing services, and with less resources. It is used for verifying code changes before pushing to production.

  • Start the staging push, by updating and pushing the stage-push branches:

    git fetch origin
git checkout stage-push
git merge --ff-only origin/master
git push
cd kumascript
git fetch origin
git checkout stage-push
git merge --ff-only origin/master
git push
cd ..

  • Prepare for testing on staging:

    • Look at the changes to be pushed (What’s Deployed on Kuma, and What’s deployed on KumaScript). To enlist the help of pull request authors and others, you can report bug numbers and PRs in IRC. firebot will give handy links to Bugzilla.

    • Think about manual tests to confirm the code changes work without errors.

    • Monitor the push in the #mdndev IRC channel. The final messages (one for kuma, one for kumascript) look like:

      🎉 SUCCESS: Check Rollout Status: Branch stage-push build #104:

  • Merge and push to the stage-integration-tests branch:

    git checkout stage-integration-tests
git merge --ff-only origin/master
git push

    This will kick off functional tests in Jenkins, which will also report to #mdndev.

  • Manually test changes on https://developer.allizom.org. Look for server errors on homepage and article pages. Try to verify features in the newly pushed code. Check the functional tests.

  • Announce in IRC that staging looks good, and you are pushing to production.

Deploy to Production

The production site is located at https://developer.mozilla.org. It is monitored by the development team and MozMEAO.

  • Pick a push song on https://www.youtube.com. Post link to IRC.

  • Start the production push:

    git fetch origin
git checkout prod-push
git merge --ff-only origin/master
git push
cd kumascript
git fetch origin
git checkout prod-push
git merge --ff-only origin/master
git push
cd ..

  • Monitor the push in the #mdndev IRC channel. The final messages (one for kuma, one for kumascript) are something like:

    🎉 SUCCESS: Check Rollout Status: Branch prod-push build #7

  • For the next 30-60 minutes,

    • Watch https://developer.mozilla.org
    • Monitor MDN in New Relic for about an hour after the push, for increased errors or performance changes.
    • Start the standby environment deployment
    • Close bugs that are now fixed by the deployment
    • Move relevant Taiga cards to Done
    • Move relevant Paper cut cards to Done

Deploy to Standby Environment

The standby environment is located in the AWS EU Frankfurt datacenter. It runs the same code and database as production, but runs in read-only maintenance mode and on minimal resources. It will be scaled up and handle MDN traffic if there is a critical failure in the AWS US West datacenter.

  • Start the standby environment push:

    git fetch origin
git checkout standby-push
git merge --ff-only origin/master
git push
cd kumascript
git fetch origin
git checkout standby-push
git merge --ff-only origin/master
git push
cd ..

  • Monitor the push in the #mdndev IRC channel. The final messages (one for kuma, one for kumascript) are something like:

    🎉 SUCCESS: Check Rollout Status: Branch standby-push build #7