CI/CD on AWS basics
Let's take a look and explore how one can implement a CI/CD workflow on AWS.
Introduction
It's 2023, so explaining CI/CD is probably not that necessary, but let's spend one sentence on the topic, so everybody's on the same page: your CI/CD (Continous Integration/ Continous Deployment) tool is connected to your source code repository and performs certain actions when something is committed, e.g.: tests and linting, creating a build, deploying to staging, integration tests, etc.
Many repository providers nowadays have CI/CD solutions included (Github, Gitlab) and there are of course many, many SaaS products that focus on CI/CD (e.g. Codeship, Travis, CircleCI, ... ). Here at Usersnap, we're big AWS users, so we usually try to use services offered by AWS.
So how does the CI/CD situation in AWS look? There are multiple services that all seem to be related to this topic:
So are you confused already? For the scope of this article, we'll focus on the services that we use, namely: CodePipeline and CodeBuild.
CodePipeline
AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
Well, it's a pipeline. You connect it to your source code, and then it starts to run through various stages (or actions) until it's done. The first stage is always a source stage, which kicks the whole thing off. Every stage has input and output artifacts, and if anything goes wrong during one of the stages, the pipeline stops.
CodeBuild
AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages.
CodeBuild is, generally speaking, like most other CI/CD services. You connect it to a source and then you get an environment in which you can run code to test, lint, build, deploy or whatever else you can think of. CodeBuild can be part of a CodePipeline, but doesn't have to – it can also run on its own.
Putting it together
CI/CD Strategy
Let's say we have a container-based web application and want to implement the following branching + CI/CD strategy.
A development branch that is always deployed to our staging environment. All development happens based on this branch.
A release branch that represents the current state of production. The development branch is merged into this branch to create a new production build, and release tags are used, to release the previously created build.
This allows us to automate deploys to both staging and production environments, while still having a manual step to release. This means, for our CI/CD process, we have the following requirements:
Any push to our repository always triggers linting + tests (except the commit starts with
[skip])Pull requests cannot be merged unless the tests + linting didn't fail
If a commit is tagged with a special tag (or on a special branch), we want to build a version of the current state on this branch and deploy this to our staging environment
If a commit is on a specific branch ("the release branch"), we want to trigger a release build
If a commit on the release branch contains a special tag, we want to trigger a release
Implementing the strategy
This totally sounds like a pipeline that does different things, depending on the branch and/or tag. So let's use CodePipeline! Since we always use CDK to setup our infrastructure, let's take a look at the CDK Api for the GitHubSourceAction:
const sourceAction = new codepipeline_actions.GitHubSourceAction({
actionName: 'GitHub_Source',
owner: 'awslabs',
repo: 'aws-cdk',
oauthToken: SecretValue.secretsManager('my-github-token'),
output: sourceOutput,
branch: 'develop', // default: 'master'
});typ
Hmmm. We want to trigger our pipeline, at least for linting and tests, for every branch – why do we need to give a branch name here? Turns out: Codepipeline's GitHubSourceAction only runs for a single tag/branch. Which means we can't use it to run tests for every new feature branch.
One solution to get out of this problem is to dynamically create a new pipeline for every branch that is created and delete it once the branch is deleted: https://aws.amazon.com/blogs/devops/multi-branch-codepipeline-strategy-with-event-driven-architecture/. But this sounds awfully complicated.
What we ended up doing, is utilizing CodeBuild like any other CI/CD provider:
Connect CodeBuild to our repository
Always run tests + linting (except the
[skip]is present in the commit)Depending on the branch/tag, build + push to the container registry, meaning the deployment has been created at this point
Depending on the branch/tag, deploy
And, in addition, have a CodePipeline for every environment (staging and production), which doesn't have GitHubSourceAction, but a EcrSourceAction, that listens for a specific container tag (e.g. release or staging) and starts the release to the corresponding environment.
const sourceAction = new codepipeline_actions.EcrSourceAction({
actionName: 'ECR',
repository: ecrRepository,
imageTag: 'staging',
output: sourceOutput,
});
In conclusion, this means we use 1 CodePipeline for every environment we're deploying to, and 1 CodeBuild project per project to do all testing + building.
Further reading
Not a lot of people seem to be doing the same as we do. The following sources pointed us into the right direction for our setup: