Nomad: Allow the use of arbitrary rkt options similar to the `java_options` config

[dansteen]

Nomad version

v0.5.6

Operating system and Environment details

Archlinux

There are many options that are able to be set on rkt. It would be nice if we could specify arbitrary options (similar to the handling of the java_options config) so we did not have to wait for each option to be supported individually by nomad.

Thanks!

[schmichael]

Are there a few options you need implemented? They're not difficult to plumb through.

We allow arbitrary config options for java because they only affect the JVM and not the container the JVM runs in. We implement containerization options on a case by case basis to try and minimize the ways in which jobs can configure themselves to escape their container. Many options also need to be handled in a special way by Nomad (such as port mappings for docker).

That being said playing whack-a-mole trying to cover all of the useful options is a constant pain for users, so I could see us implementing a workaround at least until more security features land that require locking down options.

[dansteen]

I have another ticket (https://github.com/hashicorp/nomad/issues/2629) with a couple that i need, but it would be nice to be able to use the full range of options as our needs change.

Thanks!

[blalor]

I agree with @dansteen that the full gamut would be great, but I think there's a higher-level interface to some of them that would be better for users. For example, there are a number of --insecure-options values that should be configurable without painting with the broad brush of all that is the default when trust_prefix is not specified.

Aaand I see that insecure_options is in fact an option in the driver, but not documented. :-)

[schmichael]

@blalor insecure_options was added in #2695 and will be released in 0.6 (so that's why the docs aren't live on the site yet).

Adding the thinking label here as we still have to discuss this internally. There's lots we're trying to squeeze into 0.6 already.

[onlyjob]

I think it is worth implementing to avoid specific bugs about adding new options to stay in sync with _rkt_. For instance I could use rkt_options to set --seccomp option as one of my apps does not work with current defaults.

[preetapan]

We discussed this internally yesterday. Will be closing this ticket for two reasons:

• With such a generic way to pass options, we have concerns that it allows bypassing resource constraints or other properties that are explicitly set through existing rkt driver options. There are ways we can address that in the implementation, but it didn't seem worth it.
• Our upcoming plugin work in 0.9 which will make it much easier than it is today to declare and update driver configuration. (More details to come on this in PRs as they get merged for 0.9.)

[onlyjob]

Well IMHO this ticket should be closed in favour of other viable solution and at the moment I see none... :(
I came across this due to lack of facility to pass --seccomp option. Currently no complete set of _rkt_ parameters is implemented so generic solution appears to be an easy way to cover all current and future _rkt_ options.

Why would it be a problem with resource constraints? To avoid malicious use of parameters (e.g. parameters injection)? To achieve fool proof job definitions?

Regardless, now we'll have to implement more options to match _rkt_ feature set and that means more work and more delays...