Exploring container security: four takeaways from Container Security Summit 2019

Editor’s note: On February 20, we hosted the fourth annual Container Security Summit at Google’s campus in Seattle. This event aims to help security professionals increase the security of their container deployments and apply the latest in container security research. Here’s what we learned.

Container security is a hot topic, but it can be intimidating. Container developers and operators don’t usually spend their days studying security exploits and threat analysis; likewise, container architectures and components can feel foreign to the security team.

Dev, ops, and security teams all want their workloads to be more secure (and make those pesky containers actually “contain”!); the challenge is making those teams more connected to bring container security to everyone. The theme of the 2019 Container Security Summit was just that: “More contained. More secure. More connected.” Here are four topics that led the day at the summit:

Rootless builds are here. Why aren’t you using one?

To improve the security of build processes and the isolation of running workloads, container builds should be hermetic and reproducible. A build is hermetic if no data leaks between individual builds (i.e., one build does not impact other builds), and reproducible if it’s repeatable from source to binary (i.e., you get the same output every time).

But even if your builds are hermetic and reproducible, unnecessarily running processes as root remains a potential security risk. In fact, that’s what attackers look for—how to gain privileged access to your infrastructure. “The root of all evil is unnecessarily running processes or containers as root,” said Andrew Martin, co-founder of ControlPlane, during his talk. That includes container runtimes and build tools as root.

A rootless container build doesn’t require a daemon running on the host—or ideally any root privileges at all—to build the container image. This is particularly useful when building images for untrusted workloads, such as those that come from a third-party or open-source repository that can’t be independently verified.

So, where can you get this magic? Fortunately, there are many options, including img, buildah, umoci, Kaniko, and many more! Note that with some of these, rootless container builds are optional, and some still require a daemon to be run as root, or use root inside the container. It’s still hard to get a completely rootless, unprivileged build today. Kaniko, for example, determines the minimum permissions by what’s needed to unpack your base image and execute the RUN commands. (If you’d rather not build the image yourself and trust Google’s security model, Cloud Build is a simple answer.)

Thanks to the runc work that's been ported upstream, “everybody can achieve rootlessness today,” Martin added. “No project has realised the fully-untrusted dream yet, but I expect us to reach utopia in 2019.”

The Kubernetes community showed it’s equipped to deal with vulnerabilities over the past year

Just like any other software, Kubernetes isn’t impervious to attacks. In the past year, a handful of severe vulnerabilities surfaced, including CVE-2017-1002101, which allowed containers with subpath volume mounts to access files outside the volume; and CVE-2018-1002105, which allowed a user with relatively low permissions to escalate their privileges.

Luckily, Kubernetes’ Product Security Team deftly addressed these vulnerabilities (and others) and handled the rollout of the patches. “The code is only part of the fix. When we're talking about incident response, it's not only the code, it's the process,” said CJ Cullen, software engineer on the Google Kubernetes Engine (GKE) security team.

If you’re running Kubernetes yourself, join kubernetes-announce to get the latest on releases, including vulnerability patches. If you’re running on GKE, the security bulletins will give you the latest, and let you know if there’s anything you need to do to stay safe. (Pro tip from top users: post the RSS feed in your security team’s Slack channel!)

CIS benchmarks are still the gold standard for locking down your Kubernetes configurations—but only apply what makes sense to you

The Center for Internet Security (CIS) publishes several security guidelines, including guidelines for Kubernetes. Many users refer to these guidelines to show colleagues, regulators, and customers that they’re following Kubernetes security best practices. The CIS recently updated these benchmarks for Kubernetes 1.13 (so they’re current!) and cover a wide range of recommended configurations for both the control plane and the worker nodes in your cluster.

Still, you should really think about the CIS benchmarks before you apply them. Rory McCune, Principal Consultant at NCC Group, was one of the key contributors to the Kubernetes CIS benchmarks and presented about them at the conference. “People think you should go to a benchmark and apply everything in there—but that's the wrong approach,” he said. It's important when applying any standard, to consider the environment that it's being used in, and choose which controls apply to your organization's systems.

He also explained that the CIS benchmarks are more difficult to apply to hosted solutions like GKE, “because there are many things you can’t test directly.” This creates an added step where distro users have to figure out which benchmarks apply to them, and demonstrate that to an auditor. Looking ahead, he hopes that the community will develop benchmarks for specific distributions to ease the burden on the user.

To test your current Kubernetes config against the CIS benchmarks, you can use kube-bench. In GKE, where you can’t access the control plane to test configurations, we’ve documented how we do this on your behalf in our “Control plane security” document. Best practices for GKE are laid out in the GKE hardening guide. Even with these extra steps, however, hosted solutions still offer much simpler security management than running them yourself. As Dino Dai Zovi, Staff Security Engineer at Square, said, “If you want to run your own, you're playing life on hard mode.”

We need to talk: the best way to improve container security

Container security is an evolving field; users are still finding out what works for their workloads and their priorities, but attackers wait for no one. In the unconference sessions, attendees were eager to discuss some of the issues they’ve hit with running containers securely in production, including image scanning, container isolation tools, and segmentation best practices.

The current container security landscape is still maturing, and two seemingly similar organizations might take very different approaches. The real risk, therefore, is failing to communicate across teams, said DevSecOps expert Ian Coldwater, in the closing keynote.

“Container folks, to security people, can sometimes seem like they're speaking a different language,” said Coldwater. But while container and security teams historically have failed to communicate, “every one of us has something to teach, and something to learn.” If you’re a developer running containers, be sure to keep the lines of communication to the security team wide open.

Didn’t make it to the Container Security Summit? Check out the speaker slides. You can also learn more about container security in the Exploring Container Security blog series.