Skip to main content

Deployment Process

Steps in creating and deploying a release to prod​

1. Pre-Deploy Communications​

  • Check with the Eng. Team if we are doing a Full Deploy or a Hotfix.
  • Announce it in #team-engineering and optionally discuss it in the eng standup

2. Find last Release​

;

  • On repo's main page, right side, click on Releases
  • Note tag for last release

3. Generate list of PRs for the new release​

;

  • Add /compare to repo home url, so : https://github.com/pairteam/arc-api/compare

then choose the last release by tag to compare vs main.

  • OR use 'Compare' button next to the last release. First, choose the latest tag in the pulldown. Then when you have the two "compare" sides up, choose main on the right side

  • The result should be a list of PRs, under a URL that looks something like:

https://github.com/pairteam/arc-api/compare/v2024.03.06-19.13.27-hotfix...main

  • Share the URL for the list with Eng.Team and ask:
    What should be excluded? Should any PR in main not be deployed?

  • If Yes, go to step 4, if not go to Step 6

4. Removing PRs from Release List​

Strategy #1 -- If you need to drop just one PR
  1. Revert the PR you want to exclude
  2. Do full deploy
  3. Re-apply the reverted commit so it can be included in the next deploy
Strategy #2 cut a release branch

Follow steps in #5 below

5. Creating a Hotfix or Release Branch​

  1. Create a new hotfix branch from the latest release tag
git checkout -b hotfix/v2024-03-16-hotfix v2024-03-16-1234-tag
  1. Optional: If you are not clear on the list of commits and SHAs to include, do a diff between hotfix and main:
git log hotfix/v2024-03-16-1234-hotfix… main
  1. Cherry pick commits from main
git cherry-pick abc12345678def-commit-sha
git cherry-pick ghi78912345jkl-commit-sha
  1. Compare your hotfix branch with the tag AFTER you've cherry-picked. This will help ensure you included all the commits and haven't messed it up. Also, it's useful to share that comparison with the rest of the team so everyone knows what the hotfix includes.
git log hotfix/v2024-03-16-hotfix...v2024-03-16-1234-tag
  1. Push the hotfix branch
git push -u origin hotfix/v2024-03-16-hotfix
  1. Create a DRAFT release from the hotfix branch (Deploy Process Step 7)
  2. Update the Release notes manually (Release notes are found by clicking your release under the Releases section of the main github repository page)
  3. Publish the Release (Deploy Process Step 8)
  4. Look at the build-and-deploy log scroll to see if issues as workflow completes
  5. Delete the hotfix branch
Notes:

(1) We use the 'trunk-based branching model' which means only one branch, main. All changes to hotfix branches are merged into main, so no divergence, no "cherry-pick hell". Releases are just tags to commits.

(2) We are always roll-forward on hotfixes, not roll back. (No easy way to revert a migration)

(3) If you need to merge multiple dev branches into each other, merge OLDEST branches UP into NEWER branches, finally merging into main.

6. Pre-release checks​

  1. Freeze Announcement (Announcement not necessary for hotfix releases)

When beginning release, message to Eng. team:

We are starting a deploy, please hold off merging to main.
  1. Check that Github Service is available
  • check #ntfy-external-services slack channel
  • click on link to see the GitHub Incident page, especially if Actions is not green
  • hold off till you get an all-clear from Github
  1. Are the dependencies on arc-web?
  2. Are there changes to arc-forms?
  3. Are there migrations in this release?
    • Migrations will be applied automatically after a successful deploy, before the service re-starts.

7. Create DRAFT release​

;

  1. Choose Actions tab
  2. Click on 'PROD prepare release' on left side. You'll get a list of workflow runs.

;

  1. Open the Run Workflow pop-up window on the right side. Choose:
  • From: should be Branch: main
  • Target ref should be:
    • main if full deploy
    • -- OR --
    • Specific hotfix branch from Step 5.
  1. Click Run Workflow. This creates your DRAFT Release, with notes.

8. Publish the Release​

  1. Go back to main repo page and click on Releases. Your new Draft Release should be at top.
  2. Open your Draft Release. (It should have a DRAFT chip) Then, as typical steps:
  • choose edit icon upper right
  • check 'Set as latest release'
  • choose 'Publish Release'

This will publish the release.

  1. When you want to re-deploy a past release, e.g. rollback
  • trigger 'PROD Deploy' workflow on the left
  • choose tag to deploy.

9. After Release​

  1. Check #ntfy-alerts for notifications
  2. Open up ARC in prod and check features
  3. Check if Migrations Applied
    • Go to Looker
      • SQL Runner
      • Run a statement checking your changes
  4. Notify in #team-engineering that the merge freeze is done

10. Verify in AWS (Optional)​

  1. Log onto our AWS console https://pairteam.awsapps.com/start

;

  1. Look at ECS service
    • prod is US-west-2 (oregon)
    • dev is US-east-1 (n. virginia)
  2. Make sure board is all-green
    • If issues - check logs
      • under: cluster: prod_api

11. Publish Summary of New Features​

If the new release has changed LCM/CES workflows:

  1. huddle with Max/Product Team on a communication plan
  2. create short summary of new user-facing features
  3. include screenshot or video to illustrate changes
  4. link to any external training documents or video
  5. post an update in #team-ecm channel

More guidelines in Eng. Handbook: How we Communicate.

12. Deploying to DEV​

Deployment to DEV is triggered autmatically when a PR is merged into main