It is no joke, we were involved in a great webinar on April 1st 2020. Wiebe de Roos and Keith Mokris from Palo Alto conducted a webinar in close collaboration with the CNCF. The full title of it was: CNCF Member Webinar: Container Security at Scale: Lessons Learned from the Front Lines. The webinar has been recorded and it is available on the website of the CNCF.
Introduction – the webinar
In this webinar Keith highlighted trends and challenges with regard to the adoption of containers and more specifically on container security topics. According to the CNCF, major challenges to deploy containers are still relevant: the cultural changes which are needed, container security and the complexity of this (new) deployment model. Containers are minimal since they contain only a single process, they are declarative because they are built from images which are in a human friendly format. Running containers should not be changed at all. They should do the exact same thing from run to kill.
Why container security is hard
One of the first observations is the number of containers in production. As of this year the number of containers which organizations run grows very fast. More and more organizations have > 1000 containers in production already.
In this talk, the audience will learn how to achieve this goal from a “security-centered” perspective. There will be a major focus on cloud native (network) security and container security. Key reasons why container security is a challenge for a lot of organizations:
- The number of entities is much larger compared to Virtual Machines
- Containers have a high rate of change, they are much more ephemeral
- Developers need to be aware of the security risks, not a primary domain for a lot of them
- Container security should be as portable as the containers itself, it should support containers from initial built up until running it in production, even when a container is destroyed
A practical use case
From there on, Wiebe de Roos explained how container security is practiced in an enterprise organization to keep the containers secure and compliant during their DevOps transformations. We use Prisma Cloud compute to secure their containerized workloads from birth to retire.
In the slides that follow, 5 enterprise grade challenges are presented with their solutions. A brief overview which shows the highlights:
Overcome the knowledge gap
Teams need to know where to find the correct information. It should be centered and constantly kept up to date. You should map your Prisma Cloud container security policies with detailed info as well as create real world examples and sample code. Developers should be able to experiment with it in a sandbox to learn. And last but not least: practice the “WET principle” (We Enjoy Telling). You need to constantly explain why security is a must.
Overcome the big mess of issues
A big question is: how to deal with all of the security issues and violations. Do the ultimate shift left by bringing security close to the developers: let them scan their images from the command line or directly from their IDE. The next step would be the integration pipeline. Don’t wait until a container is in production.
Use collections in Prisma Cloud to segregate teams so you are able to track issues and to prioritize which application needs attention. This is by far better than just fix all the issues which are in the reports. Don’t extend your security teams if you are overwhelmed by the number of violations: group them, hold sessions and extend your examples so you can refer to those. One key aspect here is to have a proper review process in place. If you haven’t, your organization is under high pressure and you can’t scale further.
Tips to patch images
A lot of developer face issues when they scan their container images. Instead of fixing all of the issues, adopt these tips where possible:
- Swap the container image for another one which does not have any critical vulnerabilities
- Ask your vendor to patch them or help them
- Upgrade packages and dependencies which is hard. Once successful, make those images centrally available
- Remove the insecure part (e.g. a private key) make sure you test the image
- Replace the package for another one which has the same feature (e.g. swap curl for wget)
- A final tip: perhaps containers are not the solution. Find another deployment model (e.g. serverless)
Once centralized console
To streamline efforts and to reduce overhead, only run once centralized console. Not a console per team. This releases the burden to maintain all policies and rules per team. You don’t need your central security team to have to login to all the individual dashboards to help developer teams.
But challenges are there: what if all the developer teams connect to one console and are not allowed to see each others’ findings? You need to separate them and group based upon collections.
If you have strict isolation of teams (e.g. separate networks or VPCs), find the best way to connect each other. In AWS there are multiple ways, the preferred way depends on the level of security you want: from VPC peering to private link.
Enforce and control
When you need to stay in control, keep in mind the following considerations:
What should be the “grace period” with regard to build breakers in your CI/CD pipelines. This greatly differs per organization. It can be anywhere from immediately to maybe 1 year. Find more about it in the webinar.
Who should manage the runtime defenders to protect containerized workloads on runtime systems? You need a good way to keep control of them central without having the burden to individually maintain them. It is a central part of every Kubernetes cluster. Check out the section of the container platform.
And last but not least: always find the balance between security versus business benefits. If you have optimal security which takes a while and your application misses an important deadline (say a Christmas period with high sales numbers) your business case might be ruined.
Plenty of good reasons to check out the webinar yourself. It is available on the website of the CNCF and on youtube. If you just want to scroll the slides, download them from the website of the CNCF.
See you at the next webinar!