Deploying MDN¶
MDN is served from Amazon Web Services in the US West (Oregon) datacenters. MDN is deployed as Docker containers managed by Kubernetes. The infrastructure is defined within MDN’s infra repository, including the tools used to deploy and maintain MDN.
Deploying new code to AWS takes several steps, and about 60 minutes for the full process. The deployer is responsible for emergency issues for the next 24 hours. There are 1-3 deployments per week, usually between 7 AM and 1 PM Pacific, Monday through Thursday.
Note
This page describes deployment performed by MDN staff. It requires additional setup and permissions not described here. Deployments will not work for non-staff developers, and should not be attempted.
Pre-Deployment¶
Before deploying, a staff member should:
(Once a week) Check that the latest mdn-browser-compat-data node package is included in the KumaScript image. The latest package is installed when the image is created, during the
RUN npm update
step, and can be verified by reading the build logs or running the image.(Once a week) Update the
locale
andkumascript
submodules. Thelocale
submodule must be updated to apply new translations to production. Thekumascript
submodule is not used for deployment, but updating it keeps the development environment in sync.Commit or stash any changes in your working copy.
Create a pre-push branch from master:
git fetch origin git checkout -b pre-push-`date +"%Y-%m-%d"` origin/master
Update the submodules:
git submodule update --remote git commit -m "Updating submodules" kumascript locale
Push new localizable strings. This step is only necessary when there are new strings. The TravisCI job
TOXENV=locales
checks for new strings, and you can examine the output of the recent master build to see if there are differences. At the same time, it’s not too hard to run locally and then revert if there are no useful changes.On the host system, checkout the master
locale
branch:cd locale git checkout master git pull
Update
kuma/settings/common.py
, and bump the version inPUENTE['VERSION']
.Inside the development environment, extract and rebuild the translations:
make localerefresh
On the host system (still in the
locale
subfolder), review the changes to source English strings:git diff templates/LC_MESSAGES
If there are useful changes (ignore comment lines and look for
msgid
as the changed line), commit the files in the locale submodule:git add --all . git commit
For the commit message, use the
PUENTE['VERSION']
in the commit subject, and summarize the string changes in the commit body, like:Update strings 2018.08 * Add "View All" text for document history pagination
Attempt to push to the mdn-l10n repository:
git push
If this fails, do not force with –force, or attempt to pull and create a merge commit. Someone has added a translation while you were working, and you need to start over to preserve their work:
git fetch git reset --hard @{u}
This resets your locale submodule to the new master. Start over on step 3 (
make localerefresh
).If the push to mdn-l10n is a success, commit your Kuma changes:
cd .. git commit kuma/settings/common.py locale
You can use the same commit message used for the
locale
commit.If there are no new strings, throw the changes away, starting in the
locale
folder:git checkout -- . cd .. git checkout -- kuma/settings/common.py
Push the branch and open a pull request:
git push -u origin pre-push-`date +"%Y-%m-%d"`
Check that tests pass. The
TOXENV=locales
job, which uses Dennis to check strings, is the job that occasionally fails. This is due to a string change committed in Pontoon that will break rendering in production. If this happens, fix the incorrect translation, push to master on mdn-l10n, and update the submodule in the pull request. Wait for the tests to pass.Merge the pull request. A “Rebase and merge” is preferred, to avoid a merge commit with little long-term value. Delete the pre-push branch.
Check the latest images from master are built by Jenkins (Kuma master build, KumaScript master build, both private to staff), and uploaded to the DockerHub repositories (Kuma images, KumaScript images).
Deploy to Staging¶
The staging site is located at https://developer.allizom.org. It runs on the same Kuma code as production, but against a different database, other backing services, and with less resources. It is used for verifying code changes before pushing to production.
Start the staging push, by updating and pushing the
stage-push
branches:git fetch origin git checkout stage-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout stage-push git merge --ff-only origin/master git push cd ..
Prepare for testing on staging:
Look at the changes to be pushed (What’s Deployed on Kuma, and What’s deployed on KumaScript). To enlist the help of pull request authors and others, you can report bug numbers and PRs in IRC.
firebot
will give handy links to Bugzilla.Think about manual tests to confirm the code changes work without errors.
Monitor the push in the
#mdndev
IRC channel. The final messages (one for kuma, one for kumascript) look like:🎉 SUCCESS: Check Rollout Status: Branch stage-push build #104:
Merge and push to the
stage-integration-tests
branch:git checkout stage-integration-tests git merge --ff-only origin/master git push
This will kick off functional tests in Jenkins, which will also report to
#mdndev
.Manually test changes on https://developer.allizom.org. Look for server errors on homepage and article pages. Try to verify features in the newly pushed code. Check the functional tests.
Announce in IRC that staging looks good, and you are pushing to production.
Deploy to Production¶
The production site is located at https://developer.mozilla.org. It is monitored by the development team and MozMEAO.
Pick a push song on https://www.youtube.com. Post link to IRC.
Start the production push:
git fetch origin git checkout prod-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout prod-push git merge --ff-only origin/master git push cd ..
Monitor the push in the
#mdndev
IRC channel. The final messages (one for kuma, one for kumascript) are something like:🎉 SUCCESS: Check Rollout Status: Branch prod-push build #7
For the next 30-60 minutes,
- Watch https://developer.mozilla.org
- Monitor MDN in New Relic for about an hour after the push, for increased errors or performance changes.
- Start the standby environment deployment
- Close bugs that are now fixed by the deployment
- Move relevant Taiga cards to Done
- Move relevant Paper cut cards to Done
Deploy to Standby Environment¶
The standby environment is located in the AWS EU Frankfurt datacenter. It runs the same code and database as production, but runs in read-only maintenance mode and on minimal resources. It will be scaled up and handle MDN traffic if there is a critical failure in the AWS US West datacenter.
Start the standby environment push:
git fetch origin git checkout standby-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout standby-push git merge --ff-only origin/master git push cd ..
Monitor the push in the
#mdndev
IRC channel. The final messages (one for kuma, one for kumascript) are something like:🎉 SUCCESS: Check Rollout Status: Branch standby-push build #7