This post was originally published here by amit gupta.
Using DevOps methodologies the goal is to speed up deployment using automation while increasing the predictability and manageability of the development cycle. Too often though – and this may be true more often than not – security is left for last, ultimately slowing down deployment, backing up the development pipeline, and fundamentally defeating DevOps objectives. That’s why in a DevOps environment, it’s best to bake security into every stage of the application development process right from the start.
A key DevOps principle is that the dev environment should mirror production. The same deployment automation that is used to spin up the production environment should be leveraged to spin up the dev environment, though obviously not at the same scale. Ultimately what this should mean is that something working in dev, should also work live. That way when an engineer says their code “works in their dev environment” that statement is no longer half-meaningless from a functionality standpoint. As nice as it may be to move fast and break things, you don’t want to surprise customers (and yourself) with fifty previously unknown bugs at release. This is accomplished through:
Continuous integration that incorporates unit testing, as well as some functional testing.
Continuous delivery that deploys to a qa and/or staging environment to allow for full system testing.
Deploying the application in the phoenix server pattern. Your servers should be immutable and conform to a gold standard. Or at least shouldn’t be allowed to drift over time while each one turns into a snowflake whose unique specialness makes it impossible to replicate.
For the purposes of securing both the DevOps pipeline and your build process artifacts, let’s broadly define the following four stages / security checkpoints in that pipeline:
- Development
- Build
- Pre-production Deploy
- Push to Production
The security considerations for each stage of the pipeline are heavily impacted by the type of build artifact (an application package, virtual machines, containers, etc), making it necessary to break security discussions up by build output. With that in mind, let’s start by taking a closer look at security best practices when your build output or artifacts happens to be application binaries.
Dev Sandbox
As noted earlier, the dev environment needs to mimic production with the same deployment configuration and scripts used throughout at a smaller scale. These sandbox environments should be segmented, making it impossible for them to communicate with each other, and every one of these segmented OS images needs to be hardened to the same security standards as your public facing servers.
If you don’t want snowflakes at live, then you certainly don’t want them here at the beginning of the process either, so only ever leverage gold images. Even the smallest variation is likely to cause headaches later. This is one reason why you should also create a whitelist and blacklist of external services, while also holding developers to secure code practices. Devs should not be willy-nilly doing whatever they like even when sandboxed.
Build
Install the binaries on a build sandbox, making sure when you do so that the unit and functional tests are all passing. Do they? Perfect. Now double-check that this sandbox is using the security hardened images and that this has not been updated or altered in any way by the application binary.
Could there be software vulnerabilities? You know the answer: there could always be software vulnerabilities. Audit the build sandbox to ensure there aren’t any. While you’re in an auditing mood, turn your attention to the application logs to ensure they’re compliant with the standards and requirements of your organization. Then take a fine toothed comb through the component configuration to ensure security hygiene. The last thing you want is to allow default passwords, pre-configured accounts, and clear text connection strings to slip through.
QA / Stage
New vulnerabilities may have been announced, or even worse a deployment or configuration activity may have violated a hardening standard, so repeat those audits, as you definitely don’t want to get caught napping here. Since you now have business transactions flowing through the application, check that none of the logs include PII or SPI data since an attacker could use any such data to gain deeper access. Finish up by running penetration tests. If everything looks good and you are ready for a push to production, this is the time to baseline your file system. You’ll need this baseline when monitoring the instances in production.
Push to Production
Continue your ongoing production environment monitoring, additionally looking for configuration changes, software vulnerabilities and anomalous behavior. Any access control changes, network configuration changes, unusual traffic, new processes being spawned (and the like) should be investigated immediately.
Follow these steps and you’ll help ensure that you don’t run into any unpleasant surprises down the road.
Photo:Crowdsourced Testing
