Locking Down Your Pipelines: Best Practices for Reliability and Security

CI/CD pipelines are the heart of any modern DevOps setup. They keep your software delivery flowing smoothly, like a well-oiled engine. But even the best engines need regular maintenance and some safeguards to prevent failures.  

In the world of CI/CD, those safeguards are your reliability and security practices. A poorly designed pipeline can lead to all sorts of nasty surprises: deployment errors, security vulnerabilities, and a lot of lost sleep. In this blog post, we’ll share some wisdom from the trenches, drawing from our experience to help you build and optimise rock-solid CI/CD workflows. 

Reliability: How to Build Pipelines You Can Trust 

Reliable pipelines don’t just happen. It takes careful planning and a commitment to minimise risk and ensure consistent performance. Here are some of the things we’ve learned when it comes to setting up reliable pipelines. 

Pin your actions or steps

First off, treat your actions and steps like any other piece of code: version them! Whether you’re using GitHub Actions or Azure Pipelines (we’ll go into detail later), pinning your actions and steps to specific tags offers a level of assurance. However, for maximum assurance that the action’s content remains unchanged, pinning to a commit SHA is recommended, since they are immutable. 

Use specific versions for agents/runners

When versioning, don’t (just) rely on main or latest – those are moving targets. Use fixed versions instead to guarantee that your pipeline uses the exact code you’ve tested and approved. For example, GitHub recently announced that the ubuntu-latest tag will switch from ubuntu-22.04 to ubuntu-24.04 – which could cause issues that are fully avoidable. 

Just like your actions and steps, you also need to specify the version of the runners (GitHub Actions) or agents (Azure Pipelines) that execute your pipelines. This means pinning your pipelines to use specific versions of the runner or agent image, similar to how you pin actions and steps to tags or commit SHAs. Specify the exact version you want to use. This seemingly small detail can prevent compatibility issues caused by updates to the underlying infrastructure. 

Perform reviews

Besides versioning, code reviews also aren’t just for application code. Your pipelines deserve the same level of scrutiny. Require peer reviews and approvals for all pipeline changes. This extra set of eyes can catch potential errors, ensure consistency, and enforce your team’s best practices. 

Authenticate without renewals

If you’re using GitHub Actions, consider using GitHub Apps for authentication instead of Personal Access Tokens (PATs). Unlike PATs, GitHub Apps don’t expire, saving you the hassle of constantly renewing tokens and preventing interruptions to your pipelines. Additionally, the tokens generated by GitHub Apps are less susceptible to leakage. Note that it’s critical to securely manage the private key associated with your GitHub App, as a leaked private key can pose a significant risk. 

Avoid push and pray by doing tests

For those mission-critical pipelines or custom-developed actions, think about adding unit tests. Yes, pipelines can have unit tests too! This extra layer of validation verifies that your workflows are working as intended, catching potential problems before they reach production. Besides pipelines, this gives you an extra safety net for any code or infrastructure managed within your CI/CD workflows. 

Security: Safeguarding Your CI/CD Workflows 

Your pipelines handle sensitive data and have access to your infrastructure, so keeping them secure is obviously in your best interest. 

Trust but verify

In GitHub Actions, be cautious about third-party actions. We suggest restricting usage to actions verified by GitHub, or even better, create an allow list of approved actions within your organisation or enterprise. This minimises the risk of running untrusted or potentially malicious code. If you’re a Github Enterprise user, here’s where you can set this up: 

   

Take into account the principle of least privilege

As always, keep the principle of least privilege in mind when it comes to pipeline permissions. Only grant the absolute minimum access required for your pipelines to function. Tools like the Monitor and Advisor Actions by GitHub can help you determine the appropriate level of access and keep track of their usage. 

Scan for secrets

Finally, secret scanning is your best friend for preventing sensitive information from leaking into your repositories. If you’re using Azure Pipelines, integrate secret scanning into your pipelines to automatically detect and flag any secrets that might have accidentally slipped into your code. With GitHub Advanced Security (also available for Azure DevOps), you can go a step further: you can block commits that contain secrets from entering version control altogether, preventing exposure from the outset. 

GitHub Actions vs. Azure Pipelines 

While most best practices apply across the board, GitHub Actions and Azure Pipelines have their own quirks and terminology, particularly in how they structure deployments. For example, while both platforms use the concept of environments (e.g., development, testing, production), they implement them differently. Azure Pipelines uses a higher-level concept called “stages” to group jobs, and environments are defined within these stages. In GitHub Actions, environments are typically defined at the job level, without the presence of explicit stages. 

Whatever you call them, the key is to implement control gates or checks at each stage/environment. This ensures that deployments proceed only when specific conditions are met, such as passing tests or receiving approvals. It will help prevent untested or unapproved changes from reaching your users. 

GitHub Actions tends to be more transparent with its actions, as they’re often open-source and publicly auditable. Azure DevOps extensions can be a bit more opaque, although many link to public GitHub repositories nowadays. Regardless of the platform, understanding what your actions or steps are actually doing is critical for maintaining security and reliability. Don’t just run something! 

Want a more in-depth comparison? We’ve covered the differences between GitHub and Azure DevOps in a previous blog post, “Azure DevOps vs. GitHub: Which Should You Choose?“. Give it a read if you want a deeper dive into the nuances of each platform. 

Conclusion 

Reliable and secure pipelines are essential for any successful DevOps operation. By implementing the practices we’ve discussed, you’ll build robust CI/CD workflows that you can trust.  

In the next blog post, we’ll shift our focus from security and reliability to speed, sharing efficiency tips to supercharge your pipelines and accelerate your delivery process. 

Looking to integrate GitHub Actions or Azure DevOps into your workflow? Send us a message, we’ll gladly help you out!