In this entry, I will share a real-life example of a Continuous Delivery pipeline implementation that my team and I built for a project.
Before we dive in, it is essential to understand that Continuous Delivery involves more than just setting up a pipeline on a build server.
It is a collection of practices and principles that drive software development where continuous stands for at least daily.
Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.
Jez Humble - continuousdelivery.com
The Pipeline
The pipeline is the most important piece of software we own, as it's our preferred and only way to production and we want to preserve it at all times.
The Pipeline is a crucial pattern for enabling continuous delivery. It should be customized to fit the needs of the team, and there is no one-size-fits-all approach.
Here I present a quick overview of a continuous delivery pipeline that worked well for us, presented as an inspiration for you to follow, not as a strict blueprint.
The commit stage
In the commit stage, the goal is to quickly identify a potential release candidate by checking the health of a commit and providing feedback within 10 minutes.
Here, all the checks that can be done on static code are performed, namely:
Check dependencies:
install dependencies - NPM
look for known vulnerabilities
look for noncompliant licenses based on policies/guidance
Build the Code: Make sure the code builds
Lint the Code: Although done locally, ensuring the code meets the agreed styling rules is worth reiterating.
IaC SAST: is used to perform security checks and make sure our infrastructure code is compliant with agreed security policies
Runtime SAST and Code Quality Check: To ensure our whole codebase always meets our quality standard and look for security hotspots in the runtime code
Secrets Scan: This is used to scan the commit for secrets that might have leaked through
Unit Tests: unit tests run to make sure the code behaves as expected
Publish (only on main): If all the previous checks are successful, a potential release candidate is identified leveraging Semantic Versioning and Conventional Commits, the SBOM produced in CycloneDX format, and the artifacts finalized, of which some are also made available in governance tools.
This stage of the pipeline runs on every commit, regardless of the ref of the commit (branch, MR, main trunk).
The Integration Stage
During the integration stage, our primary goal is to assess that the changes work seamlessly when deployed into the cloud. It is ensured they integrate smoothly with the existing infrastructure owned by the team.
This stage becomes particularly crucial when considering how CloudFormation works under the hood, with our project relying on it.
To verify everything integrates as intended, additional checks are carried out during the integration stage:
Deploy - The change has been deployed to the Integration environment. If there are any failures during deployment, CloudFormation will automatically roll back.
Integration API tests are carried out to ensure a service is well functioning. In these cases, the tests are carried out with every third-party service mocked out.
This stage of the pipeline runs only when a new release candidate is identified.
The Acceptance Stage
In the acceptance stage, the aim is to ensure that the release candidate can potentially be shipped.
To achieve this, additional verifications are conducted to give the team confidence to release the software.
Deploy Staging - The change is deployed in Staging, a production-like environment with configurations closely matching those of production. CloudFormation's automated rollback still applies.
API integration tests are being conducted with third-party integration enabled, which makes more sense due to the integration nature of the project, but is not without drawbacks.
Acceptance tests are conducted to ensure that the software does not regress and meets the expectations of the contract with the Product Manager / Customers.
Manual acceptance - The pipeline offers an opportunity to review changes before pushing to production. Manual review is usually not necessary.
This stage of the pipeline runs only when a new release candidate is identified.
The Release Stage
Deploy Live - If the pipeline is green, the commit is pushed to production after manual acceptance. CloudFormation's automated rollback still applies.
Tag Release - add a tag to the release commit to flag it as the release currently released in the target environment.
Monitoring - Production is constantly monitored through tools that send alerts and warnings to a dedicated Slack channel to keep its health in check, particularly after specific deployments.
This stage of the pipeline runs only when a new release candidate is identified.
The Orchestration Mechanism
Every stable environment is coordinated through an orchestration mechanism.
This orchestration mechanism acts as a semaphore: when the environment is busy with the validation other pipelines are halted, and kept waiting for the environment to free. As the environment frees, the next commit in line is dequeued and validated.
Next Steps
This pipeline architecture served me and my team well, allowing us to deploy up to 6 times a day.
Regardless, I strongly believe there is so much potential that could be unlocked, and these are most likely some next steps one could take.
Smoke tests after Prod deployment to quickly check production is working as intended after a deployment
Schema migrations - We have not had to migrate schemas so far, but we will probably have to at some point
Shift integration step on the left through ephemeral environments
Rollback button - As of today, rollback is handled through a git revert commit done by a developer, but we want to make rollback a smooth and easy operation.
Improve Security test suite - Fuzz Testing / DAST
In Numbers
Qualitative Data
Qualitative feedback is provided through the following scoring range:
1 - strongly disagree
2 - disagree
3 - neither agree nor disagree
4 - agree
5 - strongly agree
As highlighted by qualitative data, thanks to the pipeline, the team feels comfortable to release in production frequently.
“Making changes to the code base scares me”: 2 / 5
“Pushing new developments into production scares me”: 1.6 / 5
Quantitative Data
CI Feedback Time: ~9 minutes
Commit to Code Deployed Time: ~20 minutes (assuming no queue)
Pipeline success rate: 96.36% in the last month
Deploys a day: 3.1
Bonus: A word about GitOps
Except for a few specific operations as account bootstrap and data migration from a third-party system, any changes to the production environment result from a git operation.
One of the advantages of GitOps is the increased transparency around the current status of the software and the changes being made, as well as the ability to implement specific safeguards.
But there is more to it, and this link might be the right place to start from: https://www.redhat.com/en/topics/devops/what-is-gitops
Conclusion
With continuous delivery being precise on how we approach software development, the Continuous Delivery pipeline becomes a fundamental piece of software to build quality software through tiny increments.
A continuous delivery pipeline needs to be tailor-made to fit your needs, as cheap solutions might result in drama farther down the line.