Marathon: Support for "secretEnv"

[eirslett]

The REST API lets you pass environment variables into the tasks you launch, but all these variables are displayed in plaintext in the Marathon UI. It would be nice to have a property "secretEnv" in the REST API, which would behave exactly like the "env" property, but the values of these variables would not be exposed in the web UI, but like "database_password: [secret]".

Is that something that you would want to have upstream? (I could submit a patch)

[aquamatthias]

Hey @eirslett I see your point.
I fear secretEnvs would solve that specific problem but would introduce other ones.
I need to discuss how we want to deal with that topic.

In the mean time: Other users use a config database (e.g. etcd) for storing secrets.
Is that something, that would help with this?

[kamilchm]

Can you ensure (with etcd setup) that only specified Marathon app could get the password for given database?

[aquamatthias]

@kamilchm Thats one of the points I mean, when saying secretEnvs only solve a specific problem. The solution with etcd will hide sensitive data from the app definition. How the application retrieves that information is up to the application.

[kamilchm]

I think about it for a moment. I started with secretEnv as valuable thing, but then make it obsolete :) Correct me if I'm wrong.
When you look from a wider perspective of Marathon in microsevices infrastructure there are 2 places than can be used store application authentication secrets for every container.

1. Application packages defined in URIs.
2. Container environment variables, like http://12factor.net/config defines.

The first one is wrong, because it will be fetched to containers sandbox and everyone that can get to Mesos UI can see it.

The second one is better, but we need to figure out how can we get secrets populated into the container environment.

1. secretEnv - you can see it from Marathon but we can provide application view authorization in Marathon using authorization plugins from 0.12 and move auth responsibility to some authorization service.
2. use etcd like you propose - but then you need to do the same thing to authenticate your service in etcd, so you're back to secretEnv witch doesn't make sense without authz.

So I get to AuthZ finally :)

1. secretEnv without AuthZ doesn't make sense.
2. secretEnv with AuthZ doesn't make sense too, because you can just use plain env with app view authorization :)

Let's try to make https://github.com/mesosphere/marathon/pull/2233 to happen :+1:

[eirslett]

I fear secretEnvs would solve that specific problem but would introduce other ones.

What other problems do you have in mind?

The primary motivation for something like secretEnv would be that it allows you to inject secrets as environment variables - conforming to the 12-factor app spec. (So, for example, you can deploy a 3rd party application where you don't have control of the source code, and injects its DATABASE_PASSWORD as an environment variable so that it functions propertly) Even if you write all your apps yourself, it makes it easier to develop them, because you don't need logic to fetch secrets from an external service.

If we wanted an external secret backend, we could use something like vault, but then every application needs an application secret to access vault, and then these secrets must be fetched from somewhere (a chicken-and-egg problem)

And the reason why the normal env isn't enough, is that everybody who has access to Marathon can read the secrets from the UI (or the REST API). So you would need access to the server in order to read its value, not only access to the web UI. The first POST request to Marathon (in order to inject secrets) would be done by somebody/something which has access to the secrets, out of band.

1. secretEnv - you can see it from Marathon

The point would be that you _cannot_ see secretEnv values in Marathon, whether you're authenticated or not.

[kamilchm]

@eirslett I messed up too much in one post :-P
You're right with the last point and I agree with you that this can helps solve the chicken-and-egg problem :+1:

[aquamatthias]

Hey @eirslett to be more specific to the points I see:

• the person that creates/or updates the application has to supply values (which ones? are they defined somewhere? does that person know them? ) as secret env vars
• no person will see them ever again
• everybody will be able to see that values in the sandbox or can change the cmd to something like env and see all secrets on stdout
• the person that knows the secrets is not always the same person as the application developer that wants to use that

That means secretEnv is most probably only (Marathon) hiddenEnv.

We probably have to think about splitting the definition of the secrets and the usage of those.
For example think of an /v2/secrets/{bucket}/{key} endpoint, so one bucket contains a collection of key/value secrets. We could use that bucket reference in the application.
We then need #2233 to control access to secrets and access to apps.
We need a way to control the sandbox access (most probably outside of mesos)

What do you think?

[eirslett]

The way we could set up this is to let developers/users interact with another service than marathon directly for creating new services. That "deploy service" would run out of band, and know the secrets. Users would pass a Marathon-like JSON file and that service would inject tokens etc. where needed.

I don't think the mesos sandbox displays environment variables? Maybe I'm wrong?

Sure, people can change cmd to show the secrets, but that could be audited, and the barrier to doing that is higher than reading the password in plain-text. The goal isn't to make the deployment hack-proof, but rather more difficult to exploit than merely reading the password straight out of a dashboard.

A secrets REST endpoint would be nice, but we also love 12-factor apps and the principle of injecting configuration as environment variables - especially for applications you don't control the source code for. With a REST solution, one would have to make a "configuration fetch" call from within the app, which is fine if it's your own app and you know it will run on Marathon forever, but then you sacrifice portability.

Another potential use case is when you start up multiple nested Marathon instances, you have password authentication on the zookeeper instance, and don't want the zookeeper password to appear in the "parent" Marathon dashboard. (I haven't tried it myself but I imagine that's how it is today)

So the use case is basically to be able to inject secrets as environment variables and not showing the secrets in a dashboard.

[pradeepchhetri]

@aquamatthias @kamilchm @eirslett To keep things simple, I would like to suggest something which CircleCI does - showing just the last four characters of all environment variables in the marathon-ui and hide all the other characters with asterisks. This way one can verify the value of these environment variables as well and keep them hidden at the same time.

The other way, I can think of is making some standard rule like all environment variables starting with SECRET_ shouldn't be shown in marathon-ui.

Let me know what do you think.

[eirslett]

It's not just about the UI - the API would also have to be protected. But I fear that would hurt backwards-compatibility, if we just "guessed" which environment variables are secret...

[stliang]

I would like to deploy Docker containerized Apps on Mesosphere with Marathon controlling their life cycle. Vault will supply the Apps' secrets. How would one pass an Vault access token to the containerized Apps at start? I though of creating a Cubbyhole per App instance where a one time use limited life time temp token is passed to the App via Marathon's agrs field. Once the App gets its initial token, it can get a perm token for the rest of its secrets. There are some problems with this approach. The temp token tough has one time usage is visible in logs, Marathon UI, and API(s). The bigger problem is if I start 10 instances of the same App, one time usage will break. Even if I scale up one instance at a time, when the App dies, Marathon will use expired temp token. A solution might be use an unlimited usage token and rotate it using PUT API call to Marathon. I can trigger the token rotation periodically or base on Marathon's event bus where App failure event is published.

What I would prefer instead is Marathon support for Vault Cobbyhole authentication. That is Marathon provides a way to specify a secret provider such as Vault. When ever an App starts, Marathon will call this secret provider to get a secret for the App and pass the secret to it without being log or seen.

I don't think getting the initial secret is the Mesos-slave's responsibility because there could be far more Mesos-slaves than there are Marathon systems. If I can establish trust between Marathon and a secrete provider, I would prefer to propagate this trust down stream instead of establishing trust at the Mesos's layer.

I hear the Vault guys are working on an initial secret solution. If the Marathon guys are not talking to Vault guys, may be you should. We need a comprehensive solution that will also solve eirslett's security requirement.

[mikljohansson]

We had a similar problem as @stliang and @eirslett describes with the distribution of the initial secret (i.e. Vault token) and one time tokens being cumbersome in dynamic scaling envs. We didn't want the token ending up in application config files nor in the Lighter yaml config we use to drive our continuous delivery pipelines that go into Marathon. We also had some other aspects like

• Wanting to keep secrets, app config and code versions promoted together throughout or deployment pipelines. Seeing secrets as another type of app config we wanted to track all config and versions for an app in the same way, in the same place to avoid mismatches or deployment dependencies.
• Wanting to enable our very independent devteams to easily manage secrets for their services, same was as they manage the app config, versions and rollout of their services. And delegate management of what service is authorized for what secrets to devteams (with both automated checks for unencrypted secrets, and some gentle manual coaching post-commit)
• Versioning and rolling upgrades for secrets? E.g. how to roll out a new secret in a Marathon rolling upgrade? Creating and managing versioned keys in Vault seemed somewhat cumbersome.

We ended up creating Secretary which is just simple tool to allow storing encrypted secrets in config files and decrypt and inject them into the app environment at runtime, safely inside the container.

Perhaps something like that could be used to solve your initial secret distribution problem or even handle the secrets themselves until Vault has solved the initial secret problem..?

[rubbish]

@stliang We've done a little bit of work on creating a vault auth backend to authenticate a running application running on mesos/marathon: https://github.com/Banno/vault/commit/62deeb6866abd8ea95a9f2f2fe60c0f605075494

It still needs a little bit of love, but what it comes down to is that vault will check mesos and marathon when a task has asked to authenticate. It'll check the task's given remote IP against where mesos says that task is launched and makes sure that it has asked for authentication within 5 seconds of launching. In that sense we treat Mesos and Marathon as trusted authorities.

[c089]

Travis-CI uses encryption to pass sensitive data via environment variables, maybe something to consider implementing for marathon.

[ghostsquad]

Atlassian Bamboo simply obscures passwords if the variable name contains the phrase "password" by replacing the value with *******, both in the UI and in bamboo generated log output.

[jhaals]

This would be a welcome feature. We have "solved" this by putting marathon behind a reverse proxy that hides the value of all environment variables prefixed with SECRET_ in the api

[pradeepchhetri]

@jhaals Can you please explain in little detail about the reverse proxy and how you are achieving it (how do you intercept the API calls made to the marathon from this reverse proxy) please. I am also running marathon instances behind HAproxy. I would like to achieve the same.

Thanks.

[eirslett]

The proxy sounds like a nice thing to open-source! :-)

[jhaals]

It's far simpler than it might sound.
Marathon is configured to bind to localhost, apache acting reverse proxy with mod_substitute that will replace the value of SECRET_XXX with HIDDEN.

<Location "/">
    <LimitExcept GET>
        AddOutputFilterByType SUBSTITUTE application/json text/event-stream
        # replace the value of all SECRET_* keys with HIDDEN
        Substitute "s|(\"SECRET_[^\"]+\"\s*:)\s*\".*?\"|$1\"HIDDEN\"|i"
    </LimitExcept>
</Location>

Apps can therefore be updated(POST) but never viewed(GET) in the ui.
That's just how we've done it so far, if someone with more knowledge sees flaws or improvements in this setup please advice :)

[pradeepchhetri]

@jhaals Thank you Johan for letting us know the way you did it. That sounds like a nice workaround to prevent exposing the marathon secrets.

[timoreimann]

@aquamatthias @meichstedt @gkleiman The changelog mentions that Marathon 1.3 will bring some form of support for a secrets API. Is it in line with this very issue?

[jdef]

@timoreimann yes the secrets impl in 1.3 is "in line" with this issue. The API will provide a way for secrets to be specified for envvars. An operator will need to provide a plugin to actually render the value of a secret into a container at launch time. Stock Marathon 1.3 will not ship with such a plugin implementation, though it should not be difficult to write one.

For those interested in the plugin API it's here: https://github.com/mesosphere/marathon/blob/releases/1.3/plugin-interface/src/main/scala/mesosphere/marathon/plugin/task/RunSpecTaskProcessor.scala

[jdef]

@aquamatthias I think that we can close this out, yes?

[aquamatthias]

@timoreimann @jdef Yes Marathon 1.3 will support defining secrets and use them as env vars.
I created #4324 to improve our documentation on that feature.

[blackgold]

in case someone needs example implmentation to expose secrets from vault as environment variables inside the containers https://github.com/blackgold/marathon-vault-plugin

[jdef]

I visited that GH repo but didn't see any plugin source code there.

On Sun, Feb 12, 2017 at 12:38 PM, devendra ayalasomayajula <
[email protected]> wrote:

in case someone needs example implmentation to expose secrets from vault
as environment variables inside the containers
https://github.com/blackgold/marathon-vault-plugin

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mesosphere/marathon/issues/2295#issuecomment-279234413,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACPVLJb3dSRKZwwc13I6YnWs6pSz0x0jks5rb0OGgaJpZM4GBgOk
.

[blackgold]

my bad I missed adding new files. Just added now.