Faas: Feature: read-only root file-system for containers

[alexellis]

Description

A container image is immutable, but once deployed as a running container changes can be made to the file-system aka the "root file-system" mounted at /. Relates to #144

This feature implements a read-only filesystem to protect the function code and libraries from changes at run-time. I was told by @justincormack that cloud providers such as AWS Lambda do not support this use-case yet (it may have changed since).

Pros:

• enhanced security
• durable

Cons:

• could break some existing functions
• needs an additional temporary filesystem to create valid writable area

As a PoC I created a small patch for faas-netes which made all functions have a read-only file-system. This also involved mounting what Kubernetes calls an "empty-dir" to the /tmp mount-point.

Functions can use /tmp as a temporary storage area.

Scope

• Any change should be supported in Kubernetes and Docker Swarm.
• Be configurable at a per-function level via the deployment API
• Be configurable via CLI YAML

[LucasRoesler]

Copying from my comment on #144

Couple of questions about how this is used:

1. What the practical use case where a developer explicitly opts into a read-only fs or would it be opt-out to a non-root fs?
2. With the above question in mind, do we see the openfaas cluster admin having an option to enforce read-only root functions?
3. If the cluster admin can control this, does it simply override the setting and return a warning if the function is not labelled as supporting a root-only fs? In this case, the function would just break as soon as it tries to write to the fs. Or, would it just outright reject it until it is labeled correctly?
4. Do we want to set a specific convention about having a writable folder, say /opt/sandbox? Perhaps it is an ENV var that openfaas can pass in, FAAS_WRITE_DIR?

I will think a little more and come up with some more questions.

[alexellis]

The writable folder would be /tmp/ as specified. Kubernetes does not allow a hard limit on empty_dir mounts right now but @stefanprodan tells me it's coming soon.

Version 1 of this feature will be opt-in only through the YAML file / config.

Do you have enough information to start working on this?

[alexellis]

Derek add label: help wanted

[milsonian]

One issue is that in cases where a function wouldn't need a writeable filesystem at all, the lock file still implies a requirement of something like emptyDir in k8s. Maybe it would be worth removing the lockfile in of-watchdog since it has an http check? If it's tolerable to break backwards compatibility there...granted, that would imply adding something like curl for swarm, and not basing the http check in of-watchdog on said lockfile. emptyDir use is sometimes prohibited on k8s clusters via pod security policy, either due to security requirements or for general sanity (until they are manageable).

[alexellis]

This proposal suggests a read-only filesystem for the function root filesystem and a writeable temporary area using an empty-dir or similar. This accommodates a writeable area which doesn't break compatibility with the lock file.

[alexellis]

Are you interested in working on this?

[milsonian]

Probably no per the current requirements - if there's more room in terms of design/goals as things evolve, let me know though.
Cheers

[alexellis]

You can't swing it what does that mean?

[milsonian]

Edited above, to be more clear - I don't think the lockfile adds much value; I'd support an approach that allowed for both no writeable filesystems _and_ optional ones though. I have code that removes the lockfile in favor of a simple http check (for k8s) for of-watchdog and could look at what it means for swarm if/when headed that direction.

[alexellis]

We cannot make a breaking change to the interface of the system - many functions would break with no clear explanation why. Right now the healthcheck is not passed through to the function so it's impossible to indicate an unhealthy state or failure. The function can do this right now by deleting the lock file.

Having a read-only option with a writeable temporary area will allow the majority of functions to continue to operate. This is a good first step that will deliver value.

If you'd like to contribute to this project in a practical way and to have input into the design it will need flexibility and a view that respects the whole system and community of users. Thanks for sharing your suggestions.

[Templum]

I would see this as an necessary feature to operate Open FaaS in an productive environment. As this reduces potential attack vectors on functions.

[alexellis]

Since we have broad agreement the next thing is finding someone to work on this with Kubernetes being the starting point then swarm. Any volunteers?

[Templum]

I will look into it and then decide if I'm capable of implementing it

[burtonr]

@Templum how do you feel about working on this? If you'd like to tackle it, but need some help, feel free to ask in the contributors channel in Slack.
No problem if you don't want to work on this, we can look for someone else. Just let us know where you're at :smile:

[Templum]

@BurtonR sorry due to some other points I looked into, I forgot to report back :) I will start working on this matter, however as my time is limited. It will take some time, if that's fine by you

[burtonr]

Sounds good @Templum! Let us know if you need any help along the way!