ping @ewindisch @diogomonica @NathanMcCauley This is just a quick write-up. Feel free to modify/update the description if you think that's nescessary :)

thaJeztah on 26 May 2015

This is useful infos:

https://github.com/hashicorp/vault/issues/165

As is this:

https://github.com/hashicorp/vault/issues/164

dreamcat4 on 27 May 2015

@dreamcat4 there are some plans to implement a generic "secrets API", which would allow you to use either Vault, or Keywiz or you-name-it with Docker, but all in the same way. It's just an early thought, so it will require additional research.

thaJeztah on 27 May 2015

@thaJeztah Yep Sorry I don't want to detract from those efforts / discussion in any way. I am more thinking maybe it's a useful exercise also (as part of that longer process and while we are waiting) to see how far we can get right now. Then it shows up more clearly to others the limits and deficiencies in current process. What underlying is missing and needed the most to be added to improve the secrets.

Also it's worth considering about the different situations of run-time secrets VS build-time secrets. For which there is also an area overlap area.

And perhaps also (for docker) we may also be worth to consider limitations (pros/cons) between solutions that provide a mechanism to handle the secrets "in-memory". As opposed to a more heavily file-based secrets methods or network based ones e.g. local secrets server. Which are the current hacks on the table (until proper secrets API). This can help us to understand some of the unique value (for example of stronger security) added by a docker secrets API which could not otherwise be achieved by using hacks on top of the current docker feature set. However I am not a security expert. So I cannot really comment on those things with such a great certainty.

dreamcat4 on 27 May 2015

@dreamcat4 yes, you're right; for the short term, those links are indeed useful.

Also it's worth considering about the different situations of run-time secrets VS build-time secrets. For which there is also an area overlap area.

Thanks! I think I had that in my original description, must have gotten lost in the process. I will add a bullet

However I am not a security expert.

Neither am I, that's why I "pinged" the security maintainers; IMO, this should be something written by them 😇

thaJeztah on 27 May 2015

@thaJeztah great summary. I'll try to poke at this whenever I find some time.

diogomonica on 27 May 2015

@diogomonica although not _directly_ related, there a long open feature request for forwarding SSH key agent during build; https://github.com/docker/docker/issues/6396 given the number of comments, it would be good to give that some thinking too. (If even to take a decision on it whether or not it can/should be implemented)

thaJeztah on 7 Jun 2015

Assuming you could mount volumes as user other than root (I know it's impossible, but humour me), would that be a favourable approach to getting secrets into containers?

If so, I'd advocate for an alternative to -v host_dir:image_dir that expects the use of a data-only container and might look like -vc host_dir:image_dir (ie. volume-copy) wherein the contents of host_dir are copied into the image_dir volume on the data-only container.

We could then emphasize a secure-data-only containers paradigm and allow those volumes to be encrypted

ebuchman on 13 Jun 2015

I've recently read a good article about that from @jrslv where he propose to build a special docker image with secrets just to build your app, and than build another image for distribution using results from running build image.

So you have two Dockerfiles:

Dockerfile.build (here you simply copy all your secrets)
Dockerfile.dist (this one you will push to registry)

Now we can build our distribution like that:

# !/bin/sh
docker build -t hello-world-build -f Dockerfile.build .
docker run hello-world-build >build.tar.gz 
docker build -t hello-world -f Dockerfile.dist ^

Your secrets are safe, as you never push hello-world-build image.

I recommend to read @jrslv article for more details http://resources.codeship.com/ebooks/continuous-integration-continuous-delivery-with-docker

kepkin on 13 Nov 2015

👍7

Thanks for sharing @kepkin !
Just finished reading the article. Really concise!

I like the idea of exporting the files and loading them in through a separate Dockerfile. It feels like squashing without the "intermediate layers being in the build cache" issue.

However, I'm nervous that it'll complicate development and might require a third Dockerfile for simplicity.

lamroger on 13 Nov 2015

@kepkin no offense but that doesn't make any sense. Secrets are definitely not safe, since they are in the tarball and the tarball is being ADDed to production image -- even if you remove the tarball, without squashing, it will leak in some layer.

TomasTomecek on 25 Nov 2015

👎2

@TomasTomecek if I understand the example correctly, the tarball is _not_ the image-layers, but just the binary that was built inside the build container. See for example; https://github.com/docker-library/hello-world/blob/master/update.sh (no secrets involved here, but just a simple example of a build container)

thaJeztah on 25 Nov 2015

@TomasTomecek I'm talking about secrets for building Docker image. For instance, you need to pass ssh key to checkout source code from your private GitHub repository. And the tarball contains only build artifacts but doesn't contain GitHub key.

kepkin on 25 Nov 2015

@kepkin right, now I read your post again and can see it. Sorry about that. Unfortunately it doesn't solve the issue when you need secrets during deployment/building the distribution image (e.g. fetching artifacts and authenticating with artifact service). But it's definitely a good solution for separation between build process and release process.

TomasTomecek on 25 Nov 2015

@TomasTomecek that's exactly how I fetch artifacts actually.

In Docker.build image I download some binary dependencies from Amazon S3 image which require AWS key & secret. After retrieving and building, I create a tarball with everything I need.

kepkin on 25 Nov 2015

Is there a canonical "best practices" article -- the "Do"s as apprised to the "Don'ts" -- that y'all would recommend reading?

jacobdr on 27 Nov 2015

Worth noting (for anyone else like me that is stumbling upon this) that Docker Compose has support for an env_file option.

https://docs.docker.com/compose/compose-file/#env-file

afeld on 27 Nov 2015

👍4 👎1

@afeld docker itself has this feature as well, see http://docs.docker.com/engine/reference/commandline/run/#set-environment-variables-e-env-env-file but those env-vars will still show up in the same places, so don't make a difference w.r.t "leaking"

thaJeztah on 27 Nov 2015

I've stumbled across this cheat sheet: http://container-solutions.com/content/uploads/2015/06/15.06.15_DockerCheatSheet_A2.pdf

sebastian-philipp on 27 Nov 2015

❤2 👍1

@kepkin this is how I pass an ssh-key to docker build:

# serve the ssh private key once over http on a private port.
which ncat
if [ "$?" = "0" ]; then
  ncat -lp 8000 < $HOME/.ssh/id_rsa &
else
  nc -lp 8000 < $HOME/.ssh/id_rsa &
fi
nc_pid=$!
docker build --no-cache -t bob/app .
kill $nc_pid || true

and inside the Dockerfile where 172.17.0.1 is the docker gateway IP:

RUN \
  mkdir -p /root/.ssh && \
  curl -s http://172.17.0.1:8000 > /root/.ssh/id_rsa && \
  chmod 600 /root/.ssh/id_rsa && chmod 700 /root/.ssh && \
  ssh-keyscan -t rsa,dsa github.com > ~/.ssh/known_hosts && \
  git clone --depth 1 --single-branch --branch prod [email protected]/app.git . && \
  npm i --production && \
  ... && \
  rm -rf /root/.npm /root/.node-gyp /root/.ssh

If someone has something simpler let us know.

hmalphettes on 5 Dec 2015

👍11 ❤3 👎2

So what's the current status of this?

All summer there were long conversational chains, indicating just how widespread this concern is. This was filed in May, and it's still open. For instance, how would I set the password for Postgres?

jdmarshall on 8 Jan 2016

@thaJeztah What can be done to move this forward? I guess many eyes throughout different downstream projects are on this issue... ej. https://github.com/rancher/rancher/issues/1269

blaggacao on 10 Jan 2016

I guess what is being done here is kept _secret_ :D

demarant on 10 Jan 2016

😄23

This the biggest pain point for us for integrating Docker into our production stack. Is there a roadmap or another doc somewhere that points to any progress towards this ?

Chili-Man on 13 Jan 2016

Some relevant content on this topic from k8s.

gtirloni on 13 Jan 2016

What do you think of this as a potential way of addressing run-time secrets?
https://github.com/docker/docker/issues/19508

CameronGo on 20 Jan 2016

I feel like this issue would be best addressed by concentrating on a few scenarios that need to be supported, and making sure there's a set of instructions for each one. How they get implemented is less important than whether at the end of the process there's a coherent set of features that can be combined to fill the need.

A few that I've seen referred to that seem to be pretty legitimate concerns include:

Run-time Credentials

User/password information coordinated between two containers that share a link
Information is easy to keep out of your git repository
Information is easy to keep out of your pushed images (what about local containers?)
Information is easy to keep out of .bash_history (possibly a bridge too far?)
Some applications expect secrets as part of a configuration file that contains other information
Some applications expect secrets as an environment variable
Some applications allow both

When I say 'easy' I mean that there is an ergonomically sane approach to handling these variables that protects the user from accidentally doing the wrong thing and triggering a security bulletin. The stress of the experience often becomes associated with (read: blamed on) the tools involved in the mistake.

Build-time Credentials

Project is built from one or more private repositories (ex: package.json allows git urls)
Builder may be behind a password protected proxy
Builder may be using a password protected cache
End user only cares about a working image (ie, they will use pull or FROM, never docker build)
Information is easy to keep out of your pushed images

1st Edit:

Documentation of what is and is not 'leaked' into a typical image, container

What files wind up in the image (Just the COPY and ADD? anything else?)
What docker-machine retains after an image is built (especially boot2docker, but what about others?)
How environment and command line variables are captured in the image, and where they are captured
Expectations on PR issuers regarding changing these behaviors

I feel like I'm missing a couple of big ones here. Anybody got something I forgot?

jdmarshall on 21 Jan 2016

API Keys for whatever json services.

For example (and this is my real use-case), Docker build compiles a program, the API Key is necessary to authenticate me and upload the build product(s) to Bintray.com.

dreamcat4 on 21 Jan 2016

@dreamcat4 I could be way off from what you're saying, but here goes:

Are you talking about using docker images for Continuous Deployment builds, and pushing the build artifacts to an archive at the end of a successful build? Personally I prefer doing this farther up stream (eg, a post-build script in Jenkins), but if you're cross compiling that might be a bit trickier.

In my world the build agent just builds binaries/archives and retains them as 'artifacts' of the build process, and something else pushes those out to infrastructure, tags the git repository, etc. That gives me an emergency backup of the artifacts if I have a production issue and, say, my npm, docker, or Artifactory repository is down for upgrades, or the network is glitching on me.

jdmarshall on 21 Jan 2016

The point I was trying to make was about usage of API keys in general. There are many different and varied online JSON / rest services which a container may need to interact with (either at build time or run time)... which requires API keys. Does not have to be specifically about build related.

dreamcat4 on 21 Jan 2016

@dreamcat oh, so auth tokens for REST endpoints? Do you think those are handled substantially differently than, say, your postgres password in a conf file, or would you handle them similarly?

jdmarshall on 22 Jan 2016

Yeah I think those two types should be considered differently in terms of evaluating thier base minimum level of security.

API Auth tokens tend to often be:

Are not passwords
Can be revoked
Some (a much fewer subset) are single use - throw away. And essentially invalidate themselved.
Often API services are limited in their scope to just a subset of functionality. (ie read-only, or can only trigger a specific action).

Passwords tend to be / are often:

For a more fuller account access / control.
Once compromised, may be changed by attacker to something else (lock out). Or other backdoor inserted (such as db modification of other acounts held in the database, in the case of SQL).
Being a password makes a substantially much high risk of 'same password reuse' among other accounts. Wheras API keys tend to be always unique and not-usable for anything else.

So that does not necessarily mean the secrets solution _must be different_ for those 2 types. Just that the acceptable minimum baseline level of security may be a little bit lower for API keys.

This minimum level matters if having strong security is more complex / problematic to setup. (which may be true here in the case of docker secrets, or not depending how feasible / elegant the solution).

And occasionally API keys of passwords can have stronger / weaker security. Just that if one-size-fits-all is not possible.

For example - my bintray API key: that is held in same .git repo as my Dockerfile. So to keep it secure is held in PRIVATE git repo (accessed via SSH). So gaining access is to the API key relatively well protected there. However without docker having any build-in secrets functionality / protections of its own, the built docker image always includes the API key in plain text. Therefore the resulting Docker build image must be kept private like the git repository... which has a knock-on (undesirable effect) that nobody else can publically view / see the build logs / build status there.

Now that is not ideal in many ways. But the overall solution is simple enough and actually works (as in: yesterday). If there were a better mechanism made in future, I would consider switching to it. But not if that mechanism was significantly more costly / complex to setup than the current solution I have already made. So therefore extra-strong security (althogh welcome) might be overkill in the case of just 1 api key. Which just merely needs to be kept out of docker's image layers cache with some kind of a new NOCAHCE option / Dockerfile command.

Wheras a password needs something like vault or ansible-vault, and to be encrypted with yet-another-password or other strongly secure authentication mechanism. (Which we would hope not but may be a complex thing to setup).

dreamcat4 on 22 Jan 2016

I think a client/server model (like in vault) for managing and streamlining (read: auditing, break-glass) all the secrets related stuff would be good practice and would cover most of the use cases, if implementation was done thoughtfully. I, personally, am not a fan of adopting a non-holistic approach, because this is an opportunity to rise the bar in best practices.

This implies a long running client (responsibility of the person that deploys an image) and/or a build-time client (responsibility of the builder). Maybe the former one could be transferred to the docker daemon somehow which provides authorized secrets at run time.

blaggacao on 22 Jan 2016

Indeed - I wholeheartedly agree with the previous comment. Not that I don't admire the creative ways in which people are solving the problem, but I don't think this is how it needs to be - let's try and think of a solution that could be used both during CI/D and runtime, as well as taking into account that containers might be orchestrated by Mesos/Kubernetes, etc.

iangelov on 24 Jan 2016

Well, I think a bit of documentation would still be useful here, since Docker presents a few extra kinks in the problem space.

It looks like maybe the Vault guys are also looking at this from their end. I think this ticket is the one to watch:

https://github.com/hashicorp/vault/issues/165

Maybe this is something that could be collaborated upon.

jdmarshall on 25 Jan 2016

@jdmarshall

Maybe this is something that could be collaborated upon.

+1

blaggacao on 26 Jan 2016

+1 Docker + Hashi Corp Vault

thalesfsp on 28 Jan 2016

👎3

Sorry but I don't like how the solutions are getting more complex as more people pitch in. Hashi Corp Vault for instance is a full client server solution with encrypted back end storage. That adds considerable more moving parts. I'm sure some use cases demand this level of complexity but I doubt most would. If the competing solution is to use host environment variables I'm fairly sure which will end up being used by the majority of developers.

I'm looking at a solution that covers development (eg: github keys) and deployment (eg: nginx cert keys, db credentials). I don't want to pollute the host with env vars or build tools and of course no secrets should end up in github (unencrypted) or a docker image directory, even a private one.

gittycat on 8 Feb 2016

👍10

@gittycat I agree with you in the sense that there are probably several distinct use-cases. Whereby some of the solutions should be simpler than other ones.

We certainly should want to avoid resorting to ENV vars though.

My own preference is leaning towards the idea that simple key storage could be achieved with something akin to ansible's "vault" mecianism. Where you have an encrypted text file held within the build context (or sources outside / alongside the build context). Then an unlock key can unlock whatever plaintext passwords or API keys etc from that file.

I'm just saying that after using anisible's own "vault" solution. Which is relatively painless / simple. Hashicorp's vault is more secure, but it is also harder to setup and just generally more complex. Although I don't know of any technical reason why you couldn't still ultimately use that underneath as the backend (hide it / simplify it behind a docker-oriented commandline tool).

I would suggest local file storage because it avoids needing to setup some complex and potentially unreliable HTTP key storage server. Secrets storage is very much a security matter so should be available all users, not just for enterprises. Just my 2 cents opinion.

dreamcat4 on 8 Feb 2016

+1 to a local file storage backend, for more advanced use cases, I would however prefer the full power of a Hashicorp Vault - like solution. When we are talking deployment, in an organisation, the argument is, that those people who provide and control secrets are other persons than those who use them. This is a common security measure to keep the circle of persons with controlling power limited to very trusted security engineers...

blaggacao on 8 Feb 2016

👎1

Dont know if this is any use or would work, but here's a bit of a leftfield suggestion for solving the case where I want to inject a secret into a container at runtime (e.g. a postgres password)

If I could override at docker run time the entrypoint and set it to a script of my choosing e.g. /sbin/get_secrets, which after getting secrets from a mechanism of my choosing (e.g. KMS), it would exec the original entrypoint (thus becoming a mere wrapper whose sole purpose was to set up environment variables with secrets in them INSIDE the container. Such a script could be supplied at runtime via a volume mount. Such a mechanism would not involve secrets ever being written down to disk (one of my pet hates), or being leaked by docker (not part of docker inspect), but would ensure they only exist inside the environment of process 1 inside the container, which keeps the 12-factorness.

You can already do this (I believe) if entrypoint is not used in the image metadata, but only cmd is, as entrypoint then wraps the command. As mentioned the wrapper could then be mounted at runtime via a volmount. If entrypoint is already used in the image metadata, then I think you cannot accomplish this at present unless it is possible to see what the original entrypoint was from inside the container (not the cmdline override) - not sure whether you can do that or not.

Finally it would I think even be possible to supply an encrypted one-time key via traditional env var injection with which the external /sbin/get_secrets could use to request the actual secrets (e.g. the postgres password), thus adding an extra safeguard into docker leaking the one-time key.

I cant work out if this is just layers on layers, or whether it potentially solves the issue.. apologies if just the first.

gtmtech on 18 Feb 2016

👍1

@thaJeztah - I can confirm the solution I pose above works, secrets are manifested w/out being leaked by docker, they exist only in-memory for process 1 via environment variables which is perfectly 12-factor compliant, but they DO NOT show up in docker api under docker inspect, or anywhere else because they are specific to process 1. Zero work is required in the image for this to work. In my case I compiled a golang static binary to do the fetching of the secrets, so it could be volume mounted and overrode the entrypoint with this, the binary issues a sys exec to transfer control to the image-defined entrypoint when finished.

gtmtech on 19 Feb 2016

@gtmtech Interesting. Would be interested in how you found out what the original entrypoint was from your get secrets binary..

kaos on 19 Feb 2016

Maybe an example code folder would make the aproach a bit easier to demonstrate / understand.

dreamcat4 on 19 Feb 2016

Example code and working scenarios here @dreamcat4 @kaos >

https://github.com/gtmtechltd/secret-squirrel

gtmtech on 19 Feb 2016

👍1

I may be wrong, but why these complicated methods? I rely on standard unix file permissions. Hand over all secrets to docker with -v /etc/secrets/docker1:/etc/secrets readable only by root and then there's a script running at container startup as root, which passes the secrets to appropriate places for relevant programs (for example apache config). These programs drop root permissions at startup so if hacked, they cannot read the root-owned secret later. Is this method I use somehow flawed?

asokani on 19 Feb 2016

Thanks @gtmtech :)
Unfortunately, we have no standard entrypoint, nor am I able to run docker inspect prior to the docker run in a controlled manner.. But I like your approach.

kaos on 19 Feb 2016

I may be wrong, but why these complicated methods? I rely on standard unix file permissions. Hand over all secrets to docker with -v /etc/secrets/docker1:/etc/secrets readable only by root and then there's a script running at container startup as root, which passes the secrets to appropriate places for relevant programs (for example apache config). These programs drop root permissions at startup so if hacked, they cannot read the root-owned secret later. Is this method I use somehow flawed?

Hi,
I agree and think this approach ^^ should be generally recommended as best way for RUNTIME secrets. Unless anybody else here has a strong objection against that. After which can then subsequently also list any remaining corner cases (at RUNTIME) which are not covered by that ^^.

Unfortunately I can't see the secret squirrel taking off because its simply too complicated for most regular non-technical persons to learn and adopt as some popular strategy.

So then that leaves (you've probably guessed it already)...
Build-time secrets!

But I think thats a progress! Since after a long time not really getting anywhere, maybe cuts things in half and solves approx 45-50% of the total problem.

And if theres still remaining problems around secrets, at least they will be more specific / focussed ones and can keep progressing / takle afterwards.

dreamcat4 on 19 Feb 2016

👍2

Yep I wont go into too much details, but these approaches would never work for a situation I am currently working with, because I need a higher level of security than they provide. E.g. no secrets unencrypted on disk, no valid decryption keys once theyve been decrypted in the target process, regular encryption rotation, and single repository for encrypted secrets (and not spread across servers). So its more for people who have to do that level of security that I've suggested a possible approach.

secret_squirrel is anyway a hack in a space where I cant see any viable solutions yet, around docker not yet providing a secrets api, or a pluggable secrets-driver, which hopefully they will at some point, but perhaps it serves to illustrate that setting ENV vars inside the container before process exec, but not as part of docker create process (or metadata) is a secure way of being 12-factor compliant with secrets, and maybe the docker development community can use that idea when they start to build out a secrets-api/driver if they think its a good one!

Happy dockering!

gtmtech on 19 Feb 2016

We've been using the kind of approach that @gtmtech, describes, with great success. We inject KMS-encrypted secrets via environment variables, then let code inside the container decrypt as required.

Typically that involves a simple shim entrypoint, in front of the application. We currently implementing that shim with a combination of shell and a small Golang binary (https://github.com/realestate-com-au/shush), but I like the sound of the pure-Go approach.

mdub on 20 Feb 2016

👍1

@gtmtech @mdub I definitely would be pleased to see more of this.
@dreamcat4 I think the definition of "complicated" might be path dependent, which obviously is quite ok. Yet, it probably cannot be an abstractable judgment. Therefore, however, a security wrapper within the docker container doesn't seem something overly complicated to me at the design level. Another aspect is best practices: Those need to be looked at not from a developer-only perspective but from an operation perspective.
my 2 cents

blaggacao on 20 Feb 2016

Vault +1

thalesfsp on 22 Feb 2016

Vault -1. Vault has some operational characteristics (unsealing) that make it really undesirable for a lot of people.

Having a pluggable API would make the most sense.

numbsafari on 1 Mar 2016

👍1

Theres also ansible's vault. That is rather a different beast.

dreamcat4 on 1 Mar 2016

@gtmtech thanks for the suggestion, it inspired me to write this entrypoint:

#!/bin/bash

if [ -d "/var/secrets" ]; then
  tmpfile="$(mktemp)"
  for file in /var/secrets/*
  do
    if [ -f $file ]; then
      file_contents=$(cat $file)
      filename=$(basename "$file")
      underscored_filename="${filename//-/_}"
      capitalized_filename=${underscored_filename^^}
      echo "export $capitalized_filename=$file_contents" >> $tmpfile
    fi
  done

  source $tmpfile
  rm -f $tmpfile
fi

exec "$@"

I just add it into the Dockerfile like this (don't forget to chmod + x on it):

ENTRYPOINT ["/app/docker-entrypoint.sh"]

And voila. ENV vars available at runtime. Good enough :)

tomazzaman on 3 Mar 2016

👍7

If I understand correctly, the /var/secrets dir should be mounted through volumes right??
Also, when there are comment about secrets not being written to disc, how bad is write them to disc and then delete them???

blackjid on 3 Mar 2016

👍1

Nice one! You should use shred to safely delete the file though.

On Thursday, March 3, 2016, Juan Ignacio Donoso [email protected]
wrote:

If I understand correctly, the /var/secrets dir should be mounted through
volumes right??
Also, when there are comment about secrets not being written to disc, how
bad is write them to disc and then delete them???

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/13490#issuecomment-191887424.

Rui Marinho

ruimarinho on 3 Mar 2016

👍3

Inspired by @gtmtech's "secret-squirrel", I've extended my secret-management tool "shush" to make it usable as an image entry-point:

ADD shush_linux_amd64 /usr/local/bin/shush
ENTRYPOINT ["/usr/local/bin/shush", "exec", "--"]

This decrypts any KMS_ENCRYPTED_xxx envariables, and injects the results back into the environment.

https://github.com/realestate-com-au/shush#use-as-a-command-shim

mdub on 3 Mar 2016

So the thread begins with DO NOT DO ANY OF THESE THINGS.....

... but I don't see any PLEASE DO THESE THINGS INSTEAD...only various proposals/hacks that have mostly been rejected/closed.

What IS the official best-practice for now? As a docker user it's somewhat frustrating to see a long list of things we shouldn't do but then have no official alternatives offered up. Am I missing something? Does one not exist? I'm sure things are happening behind-the-scenes and that this is something that the docker team is working on, but as of right now, how do we best handle secret management until a canonical solution is presented?

alexkolson on 21 Mar 2016

👍45 ❤2

@alexkolson
As far as I understood, if you need secrets in runtime, you should either use volumes (filesystem secrets) or some services like HashiCorp Vault (network secrets).

For build-time secrets, it's more complicated.
Volumes are not supported at build time, so you should use containers to execute commands that modify filesystem, and use docker commit.

So what's missing is an ability to manage secrets on the build time using nothing except a Dockerfile, without the need to use docker commit.

Some people even say that using filesystem for secrets is not secure, and that docker daemon should provide some API to provide secrets securely (using network/firewall/automounted volume?). But nobody even have an idea of what this API would look like and how one would use it.

Vanuan on 21 Mar 2016

👍9

When I think of short comings of env vars, I think of non-docker specific issues such as:

Aggregating logs catching all the env vars or a forgotten phpinfo left on a production web server - so be careful with the secrets and config correctly.
Maybe a trojan that exposes env vars - so dont run untrusted software.
Attacks that exploit weakness such as sql injection - so validate input and use a web app firewall.

The weaknesses presented at the top this thread:

Accessible by any process in the container, thus easily "leaked"

Cross apply 1 & 2 from above. Legit but addressed with being careful right? Plus, your docker container runs far fewer processes than a full stack web server.

What about config in env var, but secret env vars have encrypted values and the app has the key in code? This is just obfuscation, because the key is in code, but would require exploits to gain access to both the key and env vars. Maybe use configuration management to manage the key on the docker host rather than in the app code. May help with rouge processes and accidental leaks but obviously not injection attacks from someone who has the key.

Preserved in intermediate layers of an image, and visible in docker inspect

Are people baking env vars in to docker images rather than setting at run time or am I misunderstanding this one. Never back secrets into artifacts right? Yes sudo docker inspect container_name gives the env var, but if your on my production server then iv already lost. sudo docker inspect image_name does not have access my env vars set at run time.

Shared with any container linked to the container

How about don't use links and the new networking instead?

The only issue that seems like a docker issue and not universal is links...

tomdavidson on 21 Mar 2016

Put me in the camp of folk who need a good way to handle secrets during docker build. We use composer for some php projects and reference some private github repos for dependencies. This means if we want to build everything inside of containers then it needs ssh keys to access these private repos.

I've not found a good and sensible way to handle this predicament without defeating some of the other things that I find beneficial about docker (see: docker squash).

I've now had to regress in building parts of the application outside of the container and using COPY to bring in the final product into the container. Meh.

I think docker build needs some functionality to handle ephemeral data like secrets so that they don't find their way into the final shipping container.

bhamilton-idexx on 23 Mar 2016

👍9

I think docker build needs some functionality to handle ephemeral data like secrets

This is a philosophical rather a technical problem. Such ephemeral data would defeat docker's essential benefit: reproducibility.

Docker's philosophy is that your Dockerfile along with a context is enough to build an image.
If you need a context to be outside of resulting image, you should fetch it from network and skip writing to filesystem. Because every Dockerfile line results in a filesystem snapshot.

If secrets should not be part of an image, you could run an ephemeral container, which would mirror/proxy all your secret-protected resources and provide secret-less access. Mirroring, btw has another rationale: https://developers.slashdot.org/story/16/03/23/0652204/how-one-dev-broke-node-and-thousands-of-projects-in-11-lines-of-javascript

You can share ssh key itself as well, but you wouldn't be able to control its usage.

Vanuan on 23 Mar 2016

@bhamilton-idexx if you make sure that the authentication to your private repositories works with a short lived token you don't have to worry about the secret being persisted in the docker image.
You have the build system generate a token with a ttl of 1 hour, make this available as an environment variable to the docker build.
You build can fetch the required build details, but the secret times out shortly after your builds completes, closing that attack vector.

wdullaer on 23 Mar 2016

Been reading a bunch of these threads now and one feature that would solve some usecases here and would have usecases outside of secrets is a --add flag for docker run that copies a file into the container, just like the ADD statement in Dockerfiles

Tebro on 24 Mar 2016

👍2

this article is A+ http://elasticcompute.io/2016/01/21/runtime-secrets-with-docker-containers/

jessfraz on 26 Mar 2016

👍15 🎉4

That is indeed a great article. Very good read. And exactly the sort of thing we have been hoping to see.

BTW:

Also found a couple of other secrets tools which seem to have been missed there from the article. Sorry for any repetitions / duplication. Didn't notice them mentioned here yet either yet:

Build time secrets:

https://github.com/defunctzombie/docket

Run time secrets:

https://github.com/ehazlett/docker-volume-libsecret

What do people think? Many thanks.

For me:

These newer tools ^^ look very good now. And they certainly didnt exist when we first started this ticket. BUT the main thing I now feel still remains missing the most:

Having a better capability for build-time secrets on the DockerHub. Which is poor there and forces an either-or choice. We must forgo the benefits of one solution for the benefits of the other one. Depending which overall set of feature(s) are more important. As local building is definately better for keeping secrets safe, but understandably worse than the Dockerhub in other ways.

dreamcat4 on 26 Mar 2016

👍9 ❤1

We've written another tool, similar to docket, that uses the new image format:

https://github.com/AngryBytes/docker-surgery

Our implementation first creates a layer containing secrets commented SECRETS, then creates a copy of the Dockerfile with a modified FROM, builds and finally removes all SECRETS layers from the resulting image.

There's always caveats to hacking this, and it'd be swell if docker had a rebasing or layer splicing functionality builtin. Removing intermediate layers right now is slow, because all solutions have to do a docker save / docker load dance behind the scenes.

Furthermore build caching is broken. Right now, we use docker commit to create the commented secrets layer, but keeping a proper cache of these layers is still a bunch of work, which we're unlikely to do. Using a Dockerfile to create the secrets layer may solve this, but there's no means of commenting the layer, making it difficult to pin-point what to remove afterwards.

stephank on 12 Apr 2016

@Vanuan [Dockerfile] can't have reproducibility. The RUN command guarantees that you and I cannot reasonably expect to get the exact same image out of two runs. Why? Because most of the time people use RUN to access network resources. If you want the same image as me you need to create your own image 'FROM' mine. No other arrangement will give us the same images. No other arrangement can give us the same images. All durable reproducibility comes from Docker Hub, not Dockerfile.

If the only defense for why we can't have ephemeral data is because Docker thinks they can remove all of the ephemeral data, then you have to deprecate the RUN instruction.

jdmarshall on 13 Apr 2016

👍6

@stephank I've implemented a docker build tool at work that takes a slightly different approach. My main concern was not for build time secrets, but it takes care of that as well (keeping the secrets out of the built image, that is, how you get hold of the secrets in the first place is still up to you).

And that is by running a "build manager" with the project code in a VOLUME. The manager then runs any number of build tools in separate containers that mount the project code using volumes from the manager. So any built artifacts and other produced files are kept in the manager volume and follows along the build pipeline for each build step. At the end, the manager can build a final production image using the produced build result. Any secrets needed along the way have been available in the manager and/or the build containers, but not the final image. No docker image wizardry used, and build caches work as expected.

What the build pipeline looks like is entirely up to the project using a spec file configuring the build requirements.

As a matter of fact, I'm rather hyped about this tool, I'm just waiting for us to be able to release it as open source (pending company policy guidelines to be adopted)..

kaos on 13 Apr 2016

👍2

@kaos On the one hand, I didn't want to deviatie from the stock Docker tooling. On the other hand, I feel like there should really be some more competition among image build tools. So that sounds interesting! 😀

stephank on 13 Apr 2016

@thaJeztah for environment (12-factor) secrets, we're locking down the Docker daemon via Twistlock (+Scalock) to prevent leakage of environment variables via inspect. Would be great if we had a Docker-native ability to not leak as much privileged info via inspect to make this a more proper reality.

bryanlatten on 22 Apr 2016

@alexkolson I think the key to this thread is "DONT DO THIS" unless you have mitigated X, Y, Z. It's clearly an engineering thead - there will always be "solutions" to common problems. That said, education on what to not do and why is important so the real workarounds can begin. The devil is always in the defaults - so we need to make sure new users understand what is at risk.

bryanlatten on 22 Apr 2016

Maybe some of you guys can help me because I don't that much experience with docker yet.
I used a Hashicorps Vault fetch my secrets.

What I basicly did was passing a token as a build argument and token can be used to fetch sensitive information from Vault. This happens at built time and can only succeed if the Vault is "unsealed" (open for fetching data) state. After building the used token is revoked.

But I think I'm still facing a few common issues.

I have to create new images if sensitive data changes
If my image got stolen / hacked then the sensitive data is within the image.

It's possible to find the used token with docker inspect but it cannot be used anymore.
I made the choice to seal and unseal hashicorps vault only at build time to limit access at the secrets store as much as possible. I also didn't saw an option to keep secrets save when fetching data at runtime.

So how bad did I do it (its ok to say if I messed up big time ;) ) does anyone have tips and tricks for me to make things more secure?

weemen on 27 Apr 2016

@weemen AFAIK storing secrets in your image is also not a good idea. Your image should have no credentials baked in (including Vault tokens). Instead, use Vault's app-id auth backend for your containers to get secrets on load time. Store them in the container's memory somehow, depending on the app stack you're using.

Also, Vault is working on an aws auth backend that will provide useful in the future if you're using AWS as a cloud provider.

jaredm4 on 27 Apr 2016

👍3

@jaredm4 Can you please clarify this statement?:

"Instead, use Vault's app-id auth backend for your containers to get secrets on load time. Store them in the container's memory somehow, depending on the app stack you're using."

I'm not yet clear on when/where to retrieve the secrets from Vault (or Keywhiz, etc). Is this done before the docker run and passed to the run command? Is this happening at some point during container initialization (if so, any examples)? Should my application retrieve these when needed? For example, my rails app needs Google API keys, do I write something inside rails to call to vault when the keys are needed?

I think I'm clear on the need for using something like Vault, and clear on how to configure it, I'm just not clear on how to consume the service and get my yml files updated amd ready when rails boots.

Any guidance here would be appreciated. Thanks

mattschutt on 6 May 2016

👍2

Sure @mcmatthew, though I must preface by saying I'm also still trying to master Vault so my experience is pretty light.

The way I have been trying to code it is that the only info you pass to the container is something needed for your code to be able to authenticate with Vault. If you're using app-id backend, that would be the app-id itself, and the address of your Vault.

On container boot, your Rails app will notice it doesn't have secrets yet, and must fetch them from Vault. It has the provided app-id, and will need to somehow generate it's user-id. This user-id generation will need to be determined by you, but their documentation hints as "it is generally a value unique to a machine, such as a MAC address or instance ID, or a value hashed from these unique values."

Once your Rails app has the app-id and user-id ready, it can then use Vault's API to /login. From there you can then make API calls to get your needed secrets.

Now to clarify what I meant about storing them in memory -- this varies depending on the type of app you're using, but with Rails there should be a way to store your secrets in a userland variable cache that will allow Rails to access the secrets from memory every request instead of getting them from Vault over and over (which as you can imagine would be slow). Take a look at this guide about caching in Rails. Namely, section 2.0, but ensuring it's using memory_cache and not disk.

Lastly, make sure that however you code it, that you do it in Rails and not with a special Docker entrypoint script or similar. Rails should detect for secrets in memory, and if not exist, fetch them.

I hope that helps. I know, a little high level, but this is how we've planned to tackle it.

jaredm4 on 6 May 2016

👍3

What's not clear is what should be kept secret, app-id, user-id or both.

Vanuan on 7 May 2016

Ok, the answer is both https://www.vaultproject.io/docs/auth/app-id.html
But it's still not clear why it any more secure than just plain firewalled access.
Maybe it's that each host secret should be tied with application (policy) secret?
I.e. if you have an access to host's secret you'd be able to access certain applications if you know their secret names?

Now we need to store 2 tokens somewhere?

Vanuan on 7 May 2016

@Vanuan They should both be kept as secret as possible, yes.

The app-id's main purpose is to restrict access to certain secrets inside Vault via Policies. Anyone with access to the app-id gains access to that app-id's policies' secrets. The app-id should be provided by your deployment strategy. For example, if using Chef, you could set it in the parameter bags (or CustomJSON for OpsWorks). However, on its own, it won't allow anyone access to Vault. So someone who gained access to Chef wouldn't then be able to then go access Vault.

The user-id is NOT provided by Chef, and should be tied to specific machines. If your app is redundantly scaled across instances, each instance should have its own user-id. It doesn't really matter where this user-id originates from (though they give suggestions), but it should not come from the same place that deployed the app-id (ie, Chef). As they said, it can be scripted, just through other means. Whatever software you use to scale instances could supply user-ids to the instances/docker containers and authorize the user-id to the app-id. It can also be done by hand if you don't dynamically scale your instances. Every time a human adds a new instance, they create a new user-id, authorize it to the app-id, and supply it to the instance via whatever means best suites them.

Is this better than firewalling instances? Guess that depends. Firewalling doesn't restrict access to secrets in Vault (afaik), and if someone gained access to your instances, they could easily enter your Vault.

This way, it's hard for them to get all the pieces of the puzzle. To take it one step further, app-id also allows for CIDR blocks which you should use. If someone somehow got the app-id and user-id, they still couldn't access Vault without being on that network.

(Again, this is my interpretation after grokking the documentation the best I could)

jaredm4 on 7 May 2016

👍4

@Vanuan @mcmatthew great questions! @jaredm4 really thanks for this clarification, this will certainly help me. This is very usefull for everyone which is looking to a more practical implementation!! If I have time some where the upcoming two weeks then Ill try again!

weemen on 7 May 2016

@thaJeztah:

Accessible by any proces in the container, thus easily "leaked"

Can you support this claim? Non-privileged processes cannot access the environment variables of non-parent processes. See https://help.ubuntu.com/community/EnvironmentVariables#Process_locality.

sanmai-NL on 18 May 2016

Environment variables set for the container (via --env or --env-file) _are_ accessible by any process in the container.

dserodio on 25 May 2016

Of course, since they are children of the entry point process. It's the job of that process, or you in case it's e.g. a shell, to unset the secret environment variables as soon as possible.

What is more relevant is whether processes with a different user ID other than 0 can access these environment variables inside and/or outside the container. This shouldn't be the case either, when the software you use inside the container properly drops privileges.

sanmai-NL on 25 May 2016

I know it's off topic but has anyone else noticed that this issue has been active for almost a full year now! Tomorrow is its anniversary. 👏

ChrisMcKenzie on 25 May 2016

🎉41 😕22 👍14

Would it be possible for a container process to read env variables in process memory and then to un-set them (in the environment) ? Does this fix most of run-time security concerns ?

davibe on 2 Jun 2016

@davibe the problem with that is that if the container or its process(es) restarts, those env vars are then gone, with no way to recover them.

BenOvermyer on 27 Jun 2016

I tried but it looks like env vars are still there after relaunch.

dade@choo:~/work/grocerest(master)$ cat test.js
console.log("FOO value: " + process.env.FOO);
delete(process.env.FOO);
console.log("FOO value after delete: " + process.env.FOO);

dade@choo:~/work/grocerest(master)$ docker run --name test -it -e FOO=BAR -v $(pwd):/data/ node node /data/test.js
FOO value: BAR
FOO value after delete: undefined

dade@choo:~/work/grocerest(master)$ docker restart test
test

dade@choo:~/work/grocerest(master)$ docker logs test
FOO value: BAR
FOO value after delete: undefined
FOO value: BAR
FOO value after delete: undefined

maybe docker-run is executing my thing as a child of bash ? I think it should not..

davibe on 28 Jun 2016

@davibe:

unset 'SECRET_ENV_VAR'

sanmai-NL on 28 Jun 2016

I think the main problem/feature in all this is that you log into Docker as root, thus anything you put inside a container can be inspected, be it a token, a volume, a variable, an encryption key... anything.

So one idea would be to remove sudo and su from your container and add a USER command before any ENTRYPOINT or CMD. Anybody running your container should now get no chance to run as root (if I'm not wrong) and thus you could now actually hide something from him.

Another idea (best IMHO) would be to add the notion of users and groups to the Docker socket and to the containers, so that you could tell GROUP-A has access to containers with TAG-B, and USER-C belongs to GROUP-A so it has access to those containers. It could even be a permission per operation (GROUP-A has access to start/stop for TAG-B, GROUP-B has access to exec, GROUP-C has access to rm/inspect, and so on).

Yajo on 12 Jul 2016

👎1

After researching this for a few hours, I cannot believe that there seems to be no officially recommended solution or workaround for build-time secrets, and something like https://github.com/dockito/vault seems to be the only viable option for build-time secrets (short of squashing the whole resulting image or building it manually in the first place). Unfortunately https://github.com/dockito/vault is quite specific to ssh keys, so off I go to try to adapt it for hosting git https credential store files as well...

NikolausDemmel on 30 Jul 2016

👍4

After what seems like forever (originally I heard it was slated for Q4 2015 release), AWS ECS seems to have finally come thru on their promise to bring IAM roles to docker apps. Here is the blog post as well.

Seems like this combined with some KMS goodness is a viable near term solution. In theory you just have to make the secrets bound to certain principals/IAM roles to keep non-auth roles from asking for something they shouldn't and leave safe storage to KMS.

Haven't tried it yet,but its on my short list...

Kubernetes also seems to have some secrets handling that reminds me a lot of Chef encrypted databags.

I understand this isn't the platform-indepentant OSS way that is the whole point of this thread,
but wanted to throw those two options out there for people playing in those infrastructure spaces who need something _NOW_

bacoboy on 30 Jul 2016

👍3 👎1

I just ran across something that might help in this regard: https://github.com/docker/docker/pull/13587

This looks like it is available starting with docker v1.10.0, but I hadn't noticed it till now. I think the solution I'm leaning toward at this point is using https://www.vaultproject.io/ to store and retrieve the secrets, storing them inside the container in a tmpfs file system mounted to /secrets or something of that nature. With the new ECS feature enabling IAM roles on containers, I believe I should be able to use vault's AWS EC2 auth to secure the authorization to the secrets themselves. (For platform independent I might be inclined to go with their App ID auth.)

In any case, the missing piece for me was where to securely put the secrets once they were retrieved. The tmpfs option seems like a good one to me. The only thing missing is that ECS doesn't seem to support this parameter yet, which is why I submitted this today: https://github.com/aws/amazon-ecs-agent/issues/469

All together that seems like a pretty comprehensive solution IMHO.

CameronGo on 3 Aug 2016

👍2

@CameronGo, thanks for the pointer. If I understand correctly this can't be used at build fine though, or can it?

NikolausDemmel on 3 Aug 2016

@NikolausDemmel sorry yes, you are correct. This is only a solution for run time secrets, not build time. In our environment, build time secrets are only used to retrieve code from Git. Jenkins handles this for us and stores the credentials for Git access. I'm not sure the same solution addresses the needs of everyone here, but I'm unclear on other use cases for build time secrets.

CameronGo on 3 Aug 2016

Jenkins handles this for us and stores the credentials for Git access.

How does that work with docker? Or do you not git clone inside the container itself?

NikolausDemmel on 6 Aug 2016

After reading through this issue in full, I believe it would benefit immensely from being split into separate issues for "build-time" and "run-time" secrets, which have very different requirements

wpalmer on 19 Aug 2016

👍30

If you are like me and you come here trying to decide what to do right now, then FWIW I'll describe the solution I settled on, until something better comes around.

For run-time secrets I decided to use http://kubernetes.io/docs/user-guide/secrets/. This only works if you use kubernetes. Otherwise vault looks ok. Anything secret either in generated image or temporary layer is a bad idea.

Regarding build-time secrets - I can't think of other build-time secrets use case other than distributing private code. At this point, I don't see better solution than relying on performing anything "secret" on the host side, and ADD the generated package/jar/wheel/repo/etc. to the image. Saving one LOC generating the package on the host side is not worth risking exposing ssh keys or complexity of running proxy server as suggested in some comments.

Maybe adding a "-v" flag to the docker build, similar to docker run flag could work well? It would temporarily share a directory between host and image, but also ensure it would appear empty in cache or in the generated image.

kozikow on 23 Aug 2016

👍4

I am currently working on a solution using Vault:

Builder machine has Vault installed and has a token saved locally
When build starts, the builder machine requests a new temporary token only valid for minutes (based on the build, 1h would even be acceptable)
Injects the token as build arg
Docker image also has Vault installed (or installs and removes it during the build) and using this token it can fetch the real secrets

It is important the the secrets are removed within the same command, so when docker caches the given layer there are no leftovers. (This of course only applies to build time secrets)

I haven't build this yet, but working on it.

sagikazarmark on 23 Aug 2016

👍2

Somewhat related to @kozikow 's comment: "Regarding build-time secrets - I can't think of other build-time secrets use case other than distributing private code."

Maybe not a build time secret specifically, but I have a use-case need for (securing) a password during build-time in a Dockerfile in order to allow for an already-built artifact to be downloaded via a RUN curl command. The build-time download requires user credentials to authenticate in order to grab the artifact - so we pass the password as an environment variable in the Dockerfile right now (we're still in Dev). Builds are happening behind the scenes automatically, as we use OpenShift, and environment variables in the Dockerfile are output to logs during the build, like any docker build command. This makes the password visible to anyone that has access to the logs, including our developers. I've been desperately trying to figure out a way to send the password so that it can be used during the docker build, but then not have the password output to logs or end up being in any layers.

I also second what @wpalmer said about breaking this thread into run-time and build-time.

cassiussa on 23 Aug 2016

I think it might be worthwhile defining some tests for whatever (runtime) secret mechanism is come up with by anyone. Because there are a lot of people on this thread that are advocating for very weak security.

As a start I suggest:

The secret does not show up in docker inspect
After process 1 has been started, the secret is not available within any file accessible from the container (including volume mounted files)
The secret is not available in /proc/1/cmdline
The secret is transmitted to the container in an encrypted form

Any solution suggested above that violates one of these is problematic.

If we can agree on a definition of what behaviour a secret should follow, then at least that will weed out endless solutions that are not fit for purpose.

gtmtech on 24 Aug 2016

👍6

@gtmtech great suggestions :)

After process 1 has been started, the secret is not available within any file accessible from the container (including volume mounted files)

I'm not sure I agree with this. While I do agree it should only be accessible from the container (in memory ideally) there are several cases where an application needs time to "start" and not have the files removed out from under it. I think something in memory for the duration of the container run (removed upon stop) is a bit better approach.

ehazlett on 24 Aug 2016

I'd add to the list of run-time requirements:

Container authentication/authorization when bootstrapping the first secret.

For instance, Vault provides for authorization with the AppRole Backend but is open-ended regarding how containers identify themselves.

Nick Sullivan presented on Clouflare's PAL project a few weeks ago, promising to open source it soon, which should provide one potential answer to the authentication question using docker notary.

ryneeverett on 24 Aug 2016

👍2

From an application's perspective there are three ways of dealing with this:

Get a secret from an environment variable.
Get a secret from a file.
Get a secret from another system.

1 and #2 above are generally the most common because the majority of applications support this mechanism. #3 is probably more ideal as it leaves the least "crumbs", but the application has to be developed specifically for this and often still has to have a credential to get the secret.

Docker is all about versatility and supporting a wide variety of use cases. On this basis 1. and 2. are most appealing from an application's view, despite the fact they both leave "crumbs" on the system.

One common approach I certainly use is to inject secrets via an entrypoint script (e.g. use a tool like credstash or plain KMS in AWS and combine with IAM roles). In this regard you actually do #3 above in the entrypoint script, and either do #1 (set an environment variable) or #2 (write to a file). This approach is dynamic and for #1 (environment variables), doesn't expose credentials in docker logs or docker inspect.

The nice thing about the entrypoint approach is you are separating the concerns of secrets management from the application.

This is an area where Docker could add functionality to avoid having to use roll your own entrypoint scripts. Docker loves plugins and could provide a hook into the container lifecycle where it could support "secrets" provider plugins, which essentially perform the function of a manual entrypoint script and inject secrets into the container (via internal environment variable or file). So you could have a Hashicorp Vault secrets provider, an AWS KMS secrets provider etc. Docker perhaps could have it's own provider based using RSA encryption (via digital certs). This whole concept is loosely similar to Kubernetes concept of secrets, which presents secrets on the container file system.

Of course there's the complexity of how do you authorize access to the secrets provider, which is a problem you face today regardless. With Hashicorp you might issue and pass a one-time/time-limited token for auth, with AWS it's IAM roles, with the Docker RSA encryption approach I mentioned, it might just be passing secrets encrypted using the Docker Engine public certificate.

mixja on 12 Sep 2016

This thread is great. I hope we see more threads like this one where people from the community and all walks of profession are able to share their experiences, thoughts and solutions.

The "secret zero" issue is a tricky one. Build-time or run-time? Both have their pro's and con's, and obvious security measures and flaws (and hacks and work arounds!).

That said, I've been thinking a lot about how the management of a pass/key comes down to the application and/or service.

Something we will be working on in the coming months is to build a shared, global configuration backing service via key/value pairs, distributed by Consul and made available as an environment variable (or injected if using environment variables is not supported). This only supports your insecure values. For secure, we will move to Vault and treat it like a backing service - much like a database or any other dependency.

Code, config and secrets will be provided via backing service(s). In this case, we use Stash, Consul and Vault. As long as the dependency is up, so is the ability to pull config and secrets as needed.

I've not seen this as a solid solution anywhere, hence I'm posting about it here. But to bring it back to the purpose of this thread, it's one approach we are going to experiment with to get around the Docker/secret issue. We will build applications which support this natively, rather than relying on the frameworks and platforms around them in which they run.

o6uoq on 12 Sep 2016

With regard to build-time secrets, Rocker's MOUNT directive has proven to be useful for creating transient directories and files that _only_ exist at build-time. Some of their templating features may also help in this situation but I haven't thoroughly used those yet.

I'd love to see this functionality implemented as a Builder plugin in Docker core (as well as some of the other useful features Rockerfiles have)!

agilgur5 on 21 Sep 2016

👍9

I see all 4 proposals currently in OP are about secret storage 🙁

I'd say dockers should facilitate passing a secret/password to a _docker instance_ but that storing/managing these secrets is (and should be) out of the scope of docker.

When _passing a secret_ i'd say a run parameter is almost perfect except being that this is usually logged. So i'd narrow this down to a non-plaintext parameter feature. An approach would be to use encryption with keys generated per docker instance.

As for _how to manage secrets_, i'd say anything that the user wants, from a homebrew bash script to integration by software like Kubernetes.

barrystaes on 18 Oct 2016

👍1

What's wrong with just implementing MOUNT like rocker mounts as @agilgur5 remarked earlier? I can't believe this debate has gone so long that a team has had to effectively fork the docker build command in order to satisfy this really easy use case. Do we need another HTTP server in the mix? KISS.

binarytemple on 18 Oct 2016

👍8

I spent so many hours around this subjet ...

For now, the best way I found to manage secrets during build phase is building within two step, so two docker files. Here a good example.

[Habitus] (http://www.habitus.io/) seems to be another option, but in my case, I do not want to add another tools mainly because I would like the build proccess on CI server AND on user's computer keep simple / same .

And what about docker-in-docker (dind) way ?

BenoitNorrin on 22 Nov 2016

Here an example of two steps build with dind like I just talked above : https://github.com/BenoitNorrin/docker-build-with-secrets

Feel free to comment ...

BenoitNorrin on 29 Nov 2016

Interesting. Kind of reminds me of how OpenShift does builds.

It looks to me like you're passing the password at the command line. Is there any way around that?

cassiussa on 29 Nov 2016

Note that there's a work-in-progress PR for build-time secrets here; https://github.com/docker/docker/pull/28079 (runtime secrets for services will be in docker 1.13, see https://github.com/docker/docker/pull/27794)

thaJeztah on 29 Nov 2016

👍4

@thaJeztah :
About the #28079, I'm a little bit pessimistic when I saw so many PR around this subject failed during the last two years ...
I doesn't want to have swarm as a dependency. Part of my customers use another cluster orchestrator.

@cassiussa:
I don't understand what you mean?
1/ Passwords was passed to the "container builder" which is not the final image. This builder do a docker build and produce an image based on the Dockerfile.realease. There are no secrets stored into this final image's history.
2/ Feel free to use docker-compose (example) if you doesn't want passing the password to the command line

BenoitNorrin on 29 Nov 2016

@BenoitNorrin I think it may be expanded to non-swarm in future, but @diogomonica may know more on that

thaJeztah on 29 Nov 2016

Sounds like it:

This is currently for Swarm mode only as the backing store is Swarm and as such is only for Linux. This is the foundation for future secret support in Docker with potential improvements such as Windows support, different backing stores, etc.

mwarkentin on 29 Nov 2016

Would be great if it were implemented the same way as rocker, can be
simple, doesn't need to be 'enterprise'.

On Tue, 29 Nov 2016, 15:53 Michael Warkentin, notifications@github.com
wrote:

Sounds like it:

This is currently for Swarm mode only as the backing store is Swarm and as
such is only for Linux. This is the foundation for future secret support in
Docker with potential improvements such as Windows support, different
backing stores, etc.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/13490#issuecomment-263608915,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZk5vlLwsBHHTTbUS_vvx-qTuwnkp6Oks5rDEpjgaJpZM4Eq021
.

binarytemple on 29 Nov 2016

👍1

A think a solution would be to encrypt some parts of information passed from a docker-compose file.

For example, run docker inspect and the encrypted information should be displayed/marked as _encrypted_. Then docker inspect --encryption-key some_key_file would show all the encrypted info, unencrypted.

On the other hand, inside the containers apps should be able to implement different mechanism to access and decrypt this encrypted info for their use.

I think encryption _is the key_ :)

hisapy on 3 Jan 2017

The purpose is for my (really, really, really common) use case is build a
software project from a git server that requires authentication, both for
the project and its dependencies. Rocker has nailed it by allowing to mount
a file or directory during build (in this case a SSH agent socket)

On Tue, 3 Jan 2017, 04:14 Hisa, notifications@github.com wrote:

A think a solution would be to encrypt some parts of information passed
from a docker-compose file.

For example, run docker inspect and the encrypted information should be
displayed/marked as encrypted. Then docker inspect --encryption-key
some_key_file would show all the encrypted info, unencrypted.

On the other hand, inside the containers apps should be able to implement
different mechanism to access and decrypt this encrypted info for their use.

I think encryption is the key :)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/13490#issuecomment-270049742,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZk5qEphZo5SR9vOWVL5dck50EPadpVks5rOcsUgaJpZM4Eq021
.

binarytemple on 3 Jan 2017

Since I didn't see it mentioned, here's another good article about handling secrets in AWS ECS: https://aws.amazon.com/blogs/security/how-to-manage-secrets-for-amazon-ec2-container-service-based-applications-by-using-amazon-s3-and-docker/

dkniffin on 12 Jan 2017

👍4

There's a new "docker secret" command in Docker 1.13. This issue should be able to be closed when the documentation for that feature is adequate to the use cases mentioned here.

vielmetti on 22 Jan 2017

👎4 👍2

The docker secret command looks to only apply currently to Docker Swarm (i.e. docker services) so is not currently viable for generic Docker containers.

mixja on 22 Jan 2017

👍5

Also docker secret only manages run time secrets, not build time secrets.

shane-axiom on 23 Jan 2017

👍3

Wow. It's like nobody on the product management team has ever considered
the use case where anything but unauthenticated open source software gets
built in a docker container or any language besides golang, where all
dependencies are copied and pasted, sorry, 'versioned' into the Git repo.

I just can't understand how folk would be so incredibly obtuse. Only
explanation I can think of is that product management team are not
practitioners and have never used the product. I often see this
characteristic manifest itself when organisation hire on the basis
jira/agile skills.

I'll just keep using rocker until 2019 or whenever someone sees sense then.

On Sun, 22 Jan 2017, 23:47 Shane StClair, notifications@github.com wrote:

Also docker secrets only manages run time secrets, not build time secrets.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/13490#issuecomment-274370450,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZk5vJVJe4OeypWd1Cwqmh8Gzyn8P-mks5rU-qqgaJpZM4Eq021
.

binarytemple on 23 Jan 2017

👎6 👍3

I take that last comment back, sorry, I'm venting. Just frustrated that
simple edge cases frequently seem like an opportunity to foist something
like consul or create something really over-engineered rather than just
implement something as simple as build time mount.

On Mon, 23 Jan 2017, 09:31 Bryan Hunt, admin@binarytemple.co.uk wrote:

Wow. It's like nobody on the product management team has ever considered
the use case where anything but unauthenticated open source software gets
built in a docker container or any language besides golang, where all
dependencies are copied and pasted, sorry, 'versioned' into the Git repo.

I just can't understand how folk would be so incredibly obtuse. Only
explanation I can think of is that product management team are not
practitioners and have never used the product. I often see this
characteristic manifest itself when organisation hire on the basis
jira/agile skills.

I'll just keep using rocker until 2019 or whenever someone sees sense
then.

On Sun, 22 Jan 2017, 23:47 Shane StClair, notifications@github.com
wrote:

Also docker secrets only manages run time secrets, not build time secrets.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/13490#issuecomment-274370450,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZk5vJVJe4OeypWd1Cwqmh8Gzyn8P-mks5rU-qqgaJpZM4Eq021
.

binarytemple on 23 Jan 2017

@binarytemple Everyone wants all the features right now. If stuff isn't ready then it's just not ready. Limiting the scope of a new feature is definitely not a bad thing as even with a limited scope there's always room for improvement.

If someone is really gung-ho about getting a feature in then they should talk to a maintainer(s) on how they can contribute the work for that.

cpuguy83 on 23 Jan 2017

I thought the same thing as @mixja in that the secret command only helps swarm users is not a more general solution (like they did with attaching persistent volumes). How you manage your secrets (what they are and who has access to them) is very system dependent and depends on which bits of paid and/or OSS you cobble together to make your "platform". With Docker the company moving into providing a platform, I'm not surprised that their first implementation is swarm based just as Hashicorp is integrating Vault into Atlas -- it makes sense.

Really how the secrets are passed falls outside the space of docker run. AWS does this kind of thing with roles and policies to grant/deny permissions plus an SDK. Chef does it using encrypted databags and crypto "bootstrapping" to auth. K8S has their own version of what just got released in 1.13. I'm sure mesos will add a similar implementation in time.

These implementations seem to fall into 2 camps.
1) pass the secret via volume mount that the "platform" provides or (chef/docker secret/k8s
2) pass credentials to talk to an external service to get things at boot (iam/credstash/etc)

I think I was hoping to see something more along the lines of the second option. In the first option, I don't think there is enough separation of concerns (the thing doing the launching also has access to all the keys), but this is preference, and like everything else in system building, everybody likes to do it different.

I'm encouraged that this first step has been taken by docker and hope that a more general mechanism for docker run comes out of this (to support camp #2) -- which sadly means I don't think this thread's initial mission has been met and shouldn't be closed yet.

bacoboy on 23 Jan 2017

👍1

like!
really simple yet very effective design

@bacoboy , @mixja - single node swarm and a single container service is not so bad
docker swarm init , docker service create replica=1

to me it is logical that docker swarm will be the default for running containers/services from now on.

krasi-georgiev on 30 Jan 2017

Am I correct in thinking that this new swarm-based proposal only impacts run-time secrets? I really don't see any need for special handling of run-time secrets, as there are already so many ways to get secrets into a running container.

build-time secrets are important, and as far as I know, this proposal does not address them.

wpalmer on 30 Jan 2017

To inject build-time secrets, we can now use docker build --squash to do the following safely:

COPY ssh_private_key_rsa /root/.ssh/id_rsa
RUN git pull ...
RUN rm -rf /root/.ssh/id_rsa

The --squash flag will produce a single layer for the entire Dockerfile: there will be no trace of the secret.

--squash is available in docker-1.13 as an experimental flag.

hmalphettes on 1 Feb 2017

@hmalphettes This means you miss out on the benefits of shared lower layers between builds.

zoidbergwill on 1 Feb 2017

👍4

This is definitely not the intention of squash. Id still be very careful about adding secrets like this.

@zoidbergwill lower layers are still shared.

cpuguy83 on 1 Feb 2017

I agree 100% with @cpuguy83. Relying on a build time flag to keep out secrets would be pretty risky. There was a proposal PR for build time (https://github.com/docker/docker/pull/30637) I'll work on a rebase to get more feedback.

ehazlett on 1 Feb 2017

👍2

@wpalmer If you have automated image builds, your tooling should know how to get build-time secrets.

For instance, you may want to keep your build-time secrets in an Ansible encrypted vault baked into an image and grant containers running from that image access to the run-time secret that keeps your vault password.

WDYT?

timka on 16 Feb 2017

Why do we keep confusing build-time secrets with runtime secrets? There are many good ways already for docker (or related tools like kubernetes) to provide the runtime secrets. The only thing really missing is build-time secrets. These secrets are not used during run time, they are used during install time, this could be internal repositories for example. The only working way I have seen in this and related topics (but also advised against it), is exposing an http server to the container during build time. The http server approach makes things quiet complicated to actually get to those secrets.

pvanderlinden on 16 Feb 2017

👍5

+1 build time secret != run time secret

As Paul points out. It is not desirable to bake internal repository
credentials into the image.

Why is this so hard to comprehend?

On Thu, 16 Feb 2017, 14:42 Paul van der Linden, notifications@github.com
wrote:

Why do we keep confusing build-time secrets with runtime secrets? There
are many good ways already for docker (or related tools like kubernetes) to
provide the runtime secrets. The only thing really missing is build-time
secrets. These secrets are not used during run time, they are used during
install time, this could be internal repositories for example. The only
working way I have seen in this and related topics (but also advised
against it), is exposing an http server to the container during build time.
The http server approach makes things quiet complicated to actually get to
those secrets.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/13490#issuecomment-280348116,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZk5h0Z2OGwApVnLNEFWKRdOfGxLOmRks5rdGBagaJpZM4Eq021
.

binarytemple on 16 Feb 2017

👍7

@pvanderlinden You can also do that with two steps building.
Here an example : https://github.com/BenoitNorrin/docker-build-with-secrets

BenoitNorrin on 16 Feb 2017

@timka as mentioned it's not desirable to bake credentials into the image as that poses a security risk. Here is a proposal for build time secrets: https://github.com/docker/docker/pull/30637

ehazlett on 16 Feb 2017

@BenoitNorrin Not sure how that would in my (and others) use case.
The packages which needs to be installed are already build when I start the docker build process. But the docker build will need to install these packages, it will need access to an internal anaconda repository and/or pypi server (python). The locations and passwords are private ofcourse.
Looks like #30637 is another attempt, hopefully this will end up in docker!

pvanderlinden on 16 Feb 2017

@timka the first half of your message seems to mention build-time secrets, but the second half explicitly talks about run-time secrets. Run-time secrets are simple. My current "solution" for build-time secrets is to pre-run, as a completely separate step, a container which fetches private data using a run-time secret. Another step then merges this into the tree, before running a regular docker build command.

The alternative, if build-time secrets were a standard feature, would be to run these steps within the Dockerfile.

My tooling does know how to run these steps automatically, but I needed to bake this myself, which is somewhat absurd for such a common desire.

wpalmer on 16 Feb 2017

FYI, I wrote https://github.com/abourget/secrets-bridge to address the build-time secrets problem.

It creates a throw-away configuration that you can pass as arguments, during the build process, it will connect to the host and fetch the secrets, use them, and then you can kill the host bridge. Even if the build-args are saved somewhere, they become useless the moment the server is killed.

The server supports SSH Agent Forwarding, tunnelled through a TLS websocket communication. It works on Windows too !

abourget on 13 Jul 2017

👍4

Alexandre, what you have done is extremely creative and skilled. It just
makes me sad that is necessary to jump thorough all these steps just to
achieve the same as could be done if 'docker build' supported the 'mount'
command rather than the blind insistence that everything be copied into the
container.

I'm my case I'm going to abandon 'docker build' and instead use rocker or
something if my own creation.

On Thu, 13 Jul 2017, 16:23 Alexandre Bourget, notifications@github.com
wrote:

FYI, I wrote https://github.com/abourget/secrets-bridge to address the
build-time secrets problem.

It creates a throw-away configuration that you can pass as arguments,
during the build process, it will connect to the host and fetch the
secrets, use them, and then you can kill the host bridge. Even if the
build-args are saved somewhere, they become useless the moment the server
is killed.

The server supports SSH Agent Forwarding, tunnelled through a TLS
websocket communication. It works on Windows too !

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/13490#issuecomment-315111388, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZk5hZqTAgPBjS9cFP_IsYNa9wv-yoAks5sNjaBgaJpZM4Eq021
.

binarytemple on 13 Jul 2017

👍1

Here is the latest secrets proposal: https://github.com/moby/moby/issues/33343

cpuguy83 on 13 Jul 2017

I think the new "multi-stage build" functionality included inside the last Docker CE release solves a big part of ours problems.

https://docs.docker.com/engine/userguide/eng-image/multistage-build/

BenoitNorrin on 13 Jul 2017

👍4

Containers spawned to execute commands from Dockerfile come with cooked /dev and no changes done there should be recorded into a layer. Can docker deliver user secrets through that mount point? Similar way it provides /dev/init?

RomanSaveljev on 8 Aug 2017

This is still pretty important because multistage builds with arguments don't leak in the image but still expose your secrets as part of process lists on the running system so its not really solved.

neclimdul on 8 Aug 2017

👍3

It's August 2017. Meanwhile "older proposals for handling secrets" in the original issue link to issues from 2014.

There are still no good solutions for build-time secrets. The PR that offered --build-time-secret flag was closed with no explanation. Nothing presented in "So, what's needed?" section is implemented.

Meanwhile

Newly installed CEO Steve Singh focusing on business customers
Latest round of $75 million to help form sales, marketing team

UPD.: as @cpuguy83 correctly and rightfully pointed out below, the full summary of proposals is at #33343

dmitriid on 21 Aug 2017

I know it's not built in, but secrets-bridge works pretty well for now.

abourget on 21 Aug 2017

@dmitriid I understand your frustration that this feature has been missing. However this is not how to address an open source community (or any community).

I posted a link to a proposal above, and have seen exactly 0 comments on it except my own.

Here is the latest secrets proposal: #33343

cpuguy83 on 21 Aug 2017

👍2

@cpuguy83 Awesome! I kinda skipped of the last third of this discussion (and a few others) as it's a lot of things to read (while at the same time looking for a solution), so I really missed your comment, sorry :(

dmitriid on 21 Aug 2017

❤1

This thread started in 2015.

It's 2017.

Why is there no solution for build-time secrets that isn't hackish and terrible, yet? It's very obviously a big issue for a lot of people, but there's still no actually good solution!

mshappe on 23 Aug 2017

👍18

@mshappe

Why is there no solution for build-time secrets that isn't hackish and terrible, yet?

Because it's a hard problem to solve correctly and it's something that will be used by literally millions of people.

Please see my comment just above yours:

I posted a link to a proposal above, and have seen exactly 0 comments on it except my own.
Here is the latest secrets proposal: #33343

If you want to see something implemented then you'll need to do more than complain that something isn't implemented. Please comment on the proposal!

cpuguy83 on 23 Aug 2017

It's so easy to solve. It just requires something, anything that isn't
baked into the images. And actually it is very easy to solve, stop using
'docker build' and use the python API, rocker, or anything else.

On Wed, Aug 23, 2017 at 9:42 PM, Brian Goff notifications@github.com
wrote:

@mshappe https://github.com/mshappe

Why is there no solution for build-time secrets that isn't hackish and
terrible, yet?

Because it's a hard problem to solve correctly and it's something that
will be used by literally millions of people.

Please see my comment just above yours:

I posted a link to a proposal above, and have seen exactly 0 comments on
it except my own.
Here is the latest secrets proposal: #33343
https://github.com/moby/moby/issues/33343

If you want to see something implemented then you'll need to do more than
complain that something isn't implemented. Please comment on the proposal!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/13490#issuecomment-324441280, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZk5oEpcipmfCji1mXz6MOVt0p6-OA6ks5sbIC0gaJpZM4Eq021
.

binarytemple on 23 Aug 2017

@binarytemple I've started looking at Rocker as an alternative, actually...but only because of this strange mental block docker seems to have about build-time secrets.

mshappe on 23 Aug 2017

It's weird. I talk to people and they're doing all kinds of stupid hacks
like using a HTTP service - throwing away everything (monitoring/granular
permissions/simplicity) the POSIX/SELinux combo provides. I just don't
understand. The refusal seems illogical to me.

On Wed, 23 Aug 2017, 23:03 Michael Scott Shappe notifications@github.com
wrote:

@binarytemple https://github.com/binarytemple I've started looking at
Rocker as an alternative, actually...but only because of this strange
mental block docker seems to have about build-time secrets.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/13490#issuecomment-324461257, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZk5ppZYsOhdfvgotCUk5l41Truo_EEks5sbJOLgaJpZM4Eq021
.

binarytemple on 24 Aug 2017

👍1

Multi-stage docker builds solves a lot of these issues.

In its simplest form you can inject secrets as build-args, and they will only be part of the image history of the images that explicitly says they need the argument. As neclimdul points out, the secrets will be available in the process listing during the build. IMO not a big issue, but we've taken another approach.

Our build server runs with some secrets mounted as volumes, so our CI copy in f.ex. /mnt/secrets/.npmrc into the current work directory. Then we use a Dockerfile simliar to the one below.

FROM node:latest
WORKDIR /usr/src/app
COPY .npmrc .
RUN echo '{ "dependencies": [ "lodash" ] }' > package.json
RUN npm install
RUN ls -lah

FROM alpine:latest
WORKDIR /usr/src/app
COPY --from=0 /usr/src/app/node_modules ./node_modules
RUN ls -lah
CMD ["ls", "./node_modules"]

The resulting image will have the installed dependencies, but not the .npmrc or any traces of it's content.

Using multi-stage builds gives you full control on how to expose build time secrets to the build process. You can get secrets from external stores like Vault, through volumes (which we mount from the Secrets store in Kubernetes), having them gpg encrypted in the repository, Travis secrets, etc.

androa on 24 Aug 2017

👍16

When using multi stage builds for this use case make sure you realize that the secret data will remain inside an untagged image in the local daemon until that image is deleted so that this data can be used for build cache in subsequent builds. But it is not pushed to the registry when pushing the final tagged image.

tonistiigi on 24 Aug 2017

👍13

@androa I like that solution but I'm not sure how I feel about the secrets being copied to the work directory. That's probably fine on a private CI server, but it's not so great for local building where you would be copying files you shouldn't out of protected locations (not to mention that the copying itself is both annoying and dangerous in that they might accidentally end up in source control). The other option would be to use a wider Docker build context, but for a lot of common secrets that could mean the whole root volume. Any suggestions on how to make this nice for local and CI?

neverfox on 1 Nov 2017

👍1

This is appalling. The self proclaimed "world’s leading software container platform" cannot be bothered to securely pass build-time secrets into containers for past 3 years now.

By "we know better" and "don't make software allowing mistakes" approach and what can be at best described as unfortunate omission at design phase, there is no support and no visible progress towards one of the required features of DevOps software. All community suggested and sometimes even developed to point of being merge-ready improvements are shut down in fear of someone abusing them. As a result of this cluster... failure, all ways to pass private keys needed only for build phase of docker container requires saving those secrets in build history, or being visible in process list with hope that, respectively, build history never leaves the trusted machine or no-one who is not supposed to ever sees the process list. Both of which will fail even most permissive security audits.

This issue is open for over 2 years now to summarize what was known about the problem then and what to do about it. There is still no solution. By that, I don't mean a comprehensive solution that will support out of the box most complex secret management schemes. There is no solution AT ALL, no host environment variables, no loading secrets from file path outside build context. Nothing that can be deemed secure in even least stringent terms.

OJezu on 14 Dec 2017

👎5 👍4

@OJezu As I have stated multiple times on this issue, there is an open proposal with basically 0 comments on it.
If you want to see secrets pushed forward, then please take the time to comment on the proposal.

Instead of coming guns blazing attacking people who work on this every day, next time try asking questions and reading at least the latest comments on the issue you are commenting on.

Things can often look stalled when really there are just people hard at work.
For build, see github.com/moby/buildkit where most of this work is happening today.

Thanks.

cpuguy83 on 14 Dec 2017

👍3

I'm a bit tipsy, because today I spent 9 hours trying to find a solution to something that should not be an issue, especially in a project that is being worked on full time and positioning itself as de facto standard and go-to soultion. I was trying very hard not to swear and stop self-harming while writing these comments.

I looked at that issue, and saw references to two solutions, one stalled since April, other already closed. I can't help but notice, that the 0 comments proposal has 4 participants, same number of comments, and mentions some apparent internal discussion, I guess by people more familiar with intricacies of . But If you want additional feedback from someone who does not even program in go, I can provide more of thoughts on problems I touched in the previous comment.

OJezu on 14 Dec 2017

@OJezu

There's at least one easy solution: use a dedicated service (e.g. run on Jenkins) to build artifacts.
That service will be securely provisioned all the required keys to access artifacts your image depends on.
On finish artifacts will be placed to a secure location (e.g. Jenkins). Those artifacts would not contain any secrets, just a directory with binaries/sources/etc.
Then another service (e.g. another Jenkins job) will access those prebuilt artifacts and turn them into an image which can be securely pushed to an image registry, which in turn is securely protected using rbac/keys to access them from developer's/production machines.

I.e. docker image build process is no different than any other build system out there: you have to have a build pipeline in place.

Vanuan on 14 Dec 2017

Build pipeline

Vanuan on 14 Dec 2017

👍3 👎1

@Vanuan not all languages are that simple with packaging and installation, to simply install with an artifact.

pvanderlinden on 15 Dec 2017

Examples?

Vanuan on 15 Dec 2017

Python projects pull in their requirements on install time, not on built time. In our case, from a private pypi/conda repository (password protected)

pvanderlinden on 15 Dec 2017

So? Make installation a part of your build process and then copy installed packages to a fresh image.

You just need to make sure that your build image and your production image are based on the same Python base image.

Vanuan on 15 Dec 2017

You can indeed just copy everything into a new image indeed. That removes the whole point of a Dockerfile though. Why have a Dockerfile if the only thing you can use it for is just to copy a set of directories?

pvanderlinden on 15 Dec 2017

So, I can't have a simple flow in which I just run docker build . wherever - either on dev machine or CI, but I have to depend on CI to build packages. Why even bother with docker then? I can write a travis file, or configure the flow in bamboo.

OJezu on 15 Dec 2017

👍2

Can't you just pip install requirements.txt in your first stage build with secrets available to pull from your private repositories. Then the next stage build just copies the site-packages from the first stage.

Why have a Dockerfile if the only thing you can use it for is just to copy a set of directories?

Why not? Using a Dockerfile to build is consistent.

oppianmatt on 15 Dec 2017

Image specification is more then just a bunch of zipped files. There are environment variables, command line arguments, volumes, etc

Read the Dockerfile reference:
https://docs.docker.com/engine/reference/builder/

It looks like you've been primarily focusing on the RUN instruction, thinking that Dockerfile is a replacement for you Makefile. It is not. Dockerfile is meant for one thing only: build an image out of some source material. What that source material would be - a binary downloaded over http or a git repository - doesn't matter. Docker doesn't need to be your CI system, even though you can use it as one under certain conditions.

Vanuan on 15 Dec 2017

I can write a travis file, or configure the flow in bamboo.

If you can get the result of your build process and then get to run it in another environment, without images and containers, then for sure you do not need to bother with docker. Why would you?

Vanuan on 15 Dec 2017

Separate, strictly controlled environment that gets guaranteed resets between builds, but only if build steps have changed. Ability to run it anywhere, not only on CI servers (like with Travis), tying build instructions with code, which I think is good if build changes for different code branches (e.g change version of run environment only on one branch). Possibility to run build container on developer machines, allowing shipping entire environment to developers who otherwise have no idea how to upgrade their own system, but will be able to build application with their changes locally with same environment as everyone else.

If I didn't want all of that, I would stick to lxc + ansible, no need for docker then.

OJezu on 15 Dec 2017

You don't need docker build for that.

Vanuan on 15 Dec 2017

You don't need docker build for that.

Of course you can provide also a custom Makefile or a build_image.sh script for every project instead of a single self-sufficient Dockerfile, but that has multiple disadvantages:

Cross platform compatibility: With providing a Dockerfile, I know that any system that can run docker build will be able to build the image. With providing a custom Makefile or build_image.sh, I have to manually ensure that those work on all platforms that I want to support.
Known interface for users: If you know docker, you know some of the behavior of docker build for any project, even without looking at the Dockerfile (e.g. with respect to caching, etc...). If I have a custom Makefile or build_image.sh, for each project, I need to first find out what are the commands to build, to clean, where and in what form the result is, if there is some caching and in what form, ...

NikolausDemmel on 15 Dec 2017

Oh, Dockerfile is far from self-sufficient. Especially for development environment.
Consider this:

most developers don't know all the different options of docker build, but almost everybody knows how to run bash scripts
docker build depends on the context directory. So unless you're willing to wait for gigabytes of data (your source code with dependencies) to travel from one location to another for every single source line change, you won't use it for development.
unless you build EVERYTHING from scratch, you have a dependency on the docker registry
it's likely that you will depend on OS repositories (whether you use Debian or Alpine-based images), unless you boot up a container straight to the statically-built binary
unless you commit everything to git, you will have some project-level dependencies, be it npm, python package index, rubygems or anything else. So you'll depend on some external package registry or its mirror
as most people noticed here you'll depend on some secret package location for your private dependencies which you can't publish to public repository, so you'll depend on that
secrets provisioning is required to access that secure location, so you'll depend on some system that will distribute secrets to developers
in addition to Dockefile, you'll need docker-compose.yml, and it's not cross-platform: you still depend on forward-/backslash differences.

Cross platform compatibility: With providing a Dockerfile, I know that any system that can run docker build will be able to build the image.

Dockerfile doesn't ensure cross-platform compatibility. You still have to provide multiple Dockerfiles for multiple platforms. "Can run docker build" doesn't mean "Uses Linux" anymore. Docker also supports Windows native images. You still have to use Cygwin + Linux VM if you want to run something specifically targeted for Linux machines on a Windows host.

Oh, and I didn't even mention x86 vs ARM...

Known interface for users: If you know docker, you know some of the behavior of docker build for any project, even without looking at the Dockerfile

Unless you don't. Everybody knows how to run a bash script without parameters or a single make command. Few people know how to correctly specify all the different command line options for docker build, docker run or docker-compose. It's inevitable that you'll have some wrapper bash or cmd script.

With all due respect to what the Docker folks did, I think you're asking too much. I'm afraid the Mobyproject doesn't have such a broad scope as to support all the development workflows imaginable.

Vanuan on 15 Dec 2017

👍3 👎1

I'm not going to refute all your points individually. Firstly, you can of course always find situations where the "single Dockerfile" approach does not work at all. However, I would argue, that for almost all of your points that you raised (which all are valid and relevant), the "custom script or makefile" approach is either just as bad or worse. Just as an example one point:

most developers don't know all the different options of docker build, but almost everybody knows how to run bash scripts

If I am involved in 10 projects, and they all use a Dockerfile, I need to learn about docker only once, but with your suggestion I need to learn 10 totally different build scripts. How do I wipe the cache and start from scratch for project Foo's build_image.sh again? It's not clear. If building the image is done with docker build, it is clear (ofc I need to know how docker works, but I also need to do that for using the image that comes out of build_image.sh).

Overall, I guess the point that me and other's are trying to make is that for /many/ scenarios the "single Dockerfile" approach seems to work really nicely for folks (which is a reason for docker being so popular), in particular in the open source world where usually all resources are accessible without secrets. But if you try to apply the same pattern that you have come to love in a context where part of your resources need credentials to access, the approach breaks down. There have been a number of suggestions (and implementations) of technologically not too complex ways to make it work, but nothing has become of it over a long time (this has been layed many times above). Hence the frustration.

I appreciate that people are putting effort into this, for example with the linked proposal in #33343. My post is about motivating what some people what and why they keep coming back asking for it here.

NikolausDemmel on 15 Dec 2017

👍2

With all due respect to what the Docker folks did, I think you're asking too much. I'm afraid the Mobyproject doesn't have such a broad scope as to support all the development workflows imaginable.

It seems to me that what most people are asking for here is nothing of the sort, but only for a simple way to use secrets in docker build in a way that is not less secure than using them in your custom build_image.sh. One way that would satisfy this need seems to be build time mounts. They have downsides, there are probably better ways, but what is being asked is not about covering every possible corner case.

NikolausDemmel on 15 Dec 2017

👍1

I'm sorry, but each person in this ticket has a slightly different use case. Those are corner cases and require different solutions.

I want to run production images on development machines. Use docker registry
I want a distributed CI system, so that each developer has a reproducible build. Use docker run to build your project, use docker prune to clean up
I want to build docker images so that I can distribute them. Use a dedicated CI server where you can run multistage builds.

Vanuan on 15 Dec 2017

👎1 👍1

@Vanuan, so I guess your approach is basically: don't use docker build, for anything more than basic environment. This is an issue created to change that. "You have to do it differently" IS the problem, not the solution.

People who push the issue want to have simpler and more straightforward approaches with docker images, not having to hack around docker limitations.

OJezu on 18 Dec 2017

👍12

For anyone interested: I had tried to exploit "masked-by-default" build-args like FTP_PROXY to build contexts. It is safe in regard to the fact that docker-build doesn't expose those masked args to image metadata nor image layers.

36443 was an attempt to expand it to a build-arg named like `SECRET` so that we can encourage users to use it as a simple work-around to the secret management problem.

However, the work has been rejected reasonably, as the masked nature of those build-args aren't guaranteed in the future.

My best bet after that is to follow @AkihiroSuda's advice, use docker build --network or a tool like habitus to store/pass secrets via a temporary tcp server only visible build contexts live within single docker daemon, at broadest.

mumoshu on 13 Mar 2018

Commenting partially, so I get notification 5 years from now, when Docker finally decides to give us a tiny step in the direction of proper credential management.... and also, to give an outline of the hack I'm using at the moment, to help others, or to get holes poked in it that I'm unaware of.

In following @mumoshu issue, I finally got the hint of using the predefined-args for build secrets.

So, essentially, I can use docker-compose, with a mapping like this:

  myProject:
    build:
      context: ../myProject/
      args: 
        - HTTPS_PROXY=${NEXUS_USERNAME}
        - NO_PROXY=${NEXUS_PASSWORD}

And then, in folder with the docker-compose.yml file, create a file named ".env" with key-value pairs of NEXUS_USERNAME and NEXUS_PASSWORD - and the proper values there.

Finally, in the Dockerfile itself, we specify our run command like so:
RUN wget --user $HTTPS_PROXY --password $NO_PROXY

And do NOT declare those as ARGs in the DockerFile.

I haven't found my credentials floating in the resulting build anywhere yet... but I don't know if I'm looking everywhere..... And for the rest of the developers on my project, they just each have to create the .env file with the proper values for them.

darmbrust on 9 Jul 2018

👍7

@darmbrust I tried your solution but couldn't make it to work.
Here is my compose yml:
version: "3.3"
services:

  buildToolsImage:
    image: vsbuildtools2017:web-v6

    build:
      context: .
      dockerfile: ./vsbuild-web-v6-optimized.dockerfile
      args:
        - CONTAINER_USER_PWD=${CONTAINER_USER_CREDS}

Here is .env file sitting next to yml file:

CONTAINER_USER_CREDS=secretpassword

And, here is my dockerfile:

# escape=`
FROM microsoft/dotnet-framework:4.7.2-sdk
# Add non-root user
CMD ["sh", "-c", "echo ${CONTAINER_USER_PWD}"] 
RUN net user userone ${CONTAINER_USER_PWD} /add /Y

And finally the command to kick this off is like this:

docker-compose -f docker-compose.buildImage.yml build

It builds the image but without using the password stored in .env file.

[Warning] One or more build-args [CONTAINER_USER_PWD] were not consumed

What am I missing here?
Thanks!

sameer-kumar on 25 Jul 2018

You have to use one of the https://docs.docker.com/engine/reference/builder/#predefined-args in the docker file. You can't use your own argument names like CONTAINER_USER_PWD.

That's how the trick works, cause docker has special behavior for the predefined-args, in that you can use them without declaring them. And by using them without declaring them, they don't appear to be logged anywhere.

With the docker-compose file, you can map those predefined args to something more reasonably named.

darmbrust on 25 Jul 2018

@darmbrust Yes, that did the trick.
However, don't you think its smelly? Any better recommendations?
Thanks!

sameer-kumar on 26 Jul 2018

It's probably not as smelly as exposing your ssh-agent credentials over tcp
via socat for any local process to steal, but indeed, as with anything
related to secrets, 'docker build' is pretty smelly indeed.

Actually, I'd forgotten that Docker for Mac can't expose Unix domain
sockets on the osx host to the containers, so that opens up even more of a
can of worms.

My current solution, run a Centos VM, GitHub machine user account
credentials go into it, build using the "Rocker" (deprecated) tool.

On Thu, 26 Jul 2018, 21:49 Sameer Kumar, notifications@github.com wrote:

@darmbrust https://github.com/darmbrust Yes, that did the trick.
However, don't you think its smelly? Any better recommendations?
Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/13490#issuecomment-408230125, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZk5iz1kvCpZ0s65ng4TwL7LmHa9zZvks5uKitDgaJpZM4Eq021
.

binarytemple on 26 Jul 2018

This entire bug is smelly. I haven't found a better way... there are several other approaches above, but I think all of the other secure ones require standing up a little http server to feed the information into the image. maybe less smelly, but more complexity, more tools, more moving parts.

Not sure that anyone has found a "good" solution... we are all stuck waiting on the docker people to do something about it... don't hold your breath, since this bug was written in 2015, and they haven't even proposed a roadmap yet, much less a solution.

darmbrust on 26 Jul 2018

It's so simple and obvious, allow the mounting of volumes, (files or
directories) into the container during the build.

This is not a technical limitation, it's a decision to not allow secrets in
order to preserve the behaviour of - check out, run build, same inputs, same
output, change a build arg if you want to invalidate the cache...

The problem is, the abstraction has becoming increasingly leaky
with people using all sorts of kludgy, insecure hacks to get a "secret"
into a container.

Newsflash, exposing your SSH keyring via TCP, even on localhost is not
secure, neither is passing credentials via environment variables (hint, run
ps, or peek in the /proc filesystem), command arguments, and environmental variables are all there, naked, for the world to see.

For developers of golang code this traditionally hasn't been an issue as they copy-and-paste
their dependencies into their projects rather than using a dependency management tool, golang developers call this practice, 'vendoring'.

For anyone working in other ecosystems where the build system
fetches dependencies from Git, or repositories that require authentication, it's a big problem.

I'm pretty sure there is some startup rule somewhere along the lines of,
"don't presume you know, how, or why, your users use the product".

On Thu, 26 Jul 2018, 22:00 Dan Armbrust, notifications@github.com wrote:

This entire bug is smelly. I haven't found a better way... there are
several other approaches above, but I think all of the other secure ones
require standing up a little http server to feed the information into the
image. maybe less smelly, but more complexity, more tools, more moving
parts.

Not sure that anyone has found a "good" solution... we are all stuck
waiting on the docker people to do something about it... don't hold your
breath, since this bug was written in 2015, and they haven't even proposed
a roadmap yet, much less a solution.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/13490#issuecomment-408233258, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZk5nvbBTj4BAv5TtELNIHhJN8mU0Ctks5uKi38gaJpZM4Eq021
.

binarytemple on 26 Jul 2018

👍2

@binarytemple Everyone who has ever worked on Docker/moby (as in the engineers behind it) know exactly what the problem is and have even run up against it.

Volumes is a solution that is itself incredibly leaky.
There is a proposal, mentioned up the comment stream a bit, that attempts to solve this in a reasonable manner (https://github.com/moby/moby/issues/33343)

The main thing here is providing the "right" abstraction rather than "any abstraction that happens to work"... we of course know this is painful for many in more than just this case.

Lots of work has been done on the builder lately which isn't necessarily visible yet, but the fruits of this effort will begin to show up in coming months.
To begin with, Docker 18.06 ships with an alternative builder implementation backed by https://github.com/moby/buildkit.
You may think "how does this help me?". Buildkit provides a lot of low-level primitives that enables us to be much more flexible in the Docker builder. Even so much as to be able to provide your own build parser (which can be anything from an enhanced Dockerfile parser to something completely different). Parsers are specified at the top of the "Dockerfile" and are just any image you want to use to parse the file.

If you really want to see something right now, you can take buildkit itself and run with it today, it sits on top of containerd, you can build a custom integration pretty quickly.

cpuguy83 on 27 Jul 2018

Secret mounts support was added to buildkit in https://github.com/moby/buildkit/pull/522 . They appear strictly on tmpfs, are excluded from build cache and can use a configurable data source. No PR yet that exposes it in a dockerfile syntax but should be a simple addition.

tonistiigi on 27 Jul 2018

❤7

There are 2 solutions to build images with secrets.

Multi-stage build :

FROM ubuntu as intermediate
ARG USERNAME
ARG PASSWORD
RUN git clone https://${USERNAME}:${PASSWORD}@github.com/username/repository.git

FROM ubuntu
# copy the repository form the previous image
COPY --from=intermediate /your-repo /srv/your-repo

Then : docker build --build-arg USERNAME=username --build-arg PASSWORD=password my-image .

Using a image builder : docker-build-with-secrets

BenoitNorrin on 27 Jul 2018

👍2

@BenoitNorrin sorry, but you've exposed that password to every process on the host system. Unix security 101 - don't put secrets as command arguments.

binarytemple on 27 Jul 2018

Yes but there are some usages where security matters a little less:

you want to build on your own computer
you build on your entreprise CI server (like jenkins). Most of the time it's about having access to a private repository (nexus, git, npm, etc), so your CI may have her own credentials for that.
you can use a VM created from docker-machine and remove it after.

BenoitNorrin on 27 Jul 2018

👍1

If that's the only problem, @binarytemple, then simply adding the flag docker image build --args-file ./my-secret-file should be a pretty easy fix for this whole problem, isn't it? :thinking:

Yajo on 27 Jul 2018

👍1

@yajo could be, yes it's at least a workaround until buildkit ships with secrets mount. Good suggestion. Thanks. B

binarytemple on 27 Jul 2018

👍1

Unfortunately most of the workarounds mentioned in these and the many other tickets still expose the secrets to the resulting image, or only works with specific languages where you only need dependencies during compile time and not during installation.

@binarytemple that will never happen, the docker maintainers have already killed at least one PR fully documented and fully implemented of a safe secret feature. Given the rest of history (this 3 year old ticket isn't the oldest and definitely not the only ticket/PR on this topic) I think it's safe to say the docker maintainers don't understand the need for security, which is a big problem.

pvanderlinden on 27 Jul 2018

😕1

The biggest pain point is secret rotations for me

you've to maintain a graph of secret to services dependencies, and update twice each service (to get back to original secret name)

listing secrets from services doesn't seem to be easy (I gave up after some attempts around docker service inspect --format='{{.Spec.TaskTemplate.ContainerSpec.Secrets}}' <some_service>), listing services dependencies from docker secret inspect <secret_name> would be useful too. So I just maintain that (approximate) graph manually for now.

You also have to specify the secret destination, when it's not the default /run/secrets/<secret_name> in the docker service update command

I just hope for a simpler way to rotate secrets

caub on 14 Aug 2018

@caub here's some CLI help:

Docker docs for formatting help come up with the rest of your inspect format:

docker service inspect --format='{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{println .SecretName}}{{end}}'

That'll list all secret names in a service. If you wanted both name and ID, you could:

docker service inspect --format='{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{println .SecretName .SecretID}}{{end}}' nginx

I always have my CI/CD (service update commands) or stack files hardcode the path so you don't have that issue on rotation.

With labels you can have CI/CD automation identify the right secret if you're not using stack files (without needing the secret name, which would be different each time).

BretFisher on 14 Aug 2018

👍1

docker build --secret is finally available in Docker 18.09 https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

@thaJeztah Are we ready to close this issue?

AkihiroSuda on 11 Nov 2018

🎉5

For older versions of docker, using multistage build with copying secrets before build command is viable option, right?

```
FROM debian as build
COPY ./secret.conf /path/on/image/
RUN build.sh
...

FROM debian
COPY --from=build ...

andriy-f on 11 Nov 2018

@andriy-f yes, that works, as long as you;

(obviously) don't copy the secret to the final stage 😉, or:
use the build stage / stage in which a secret is present as a "parent" for the final image
never _push_ the build-stage to a registry
trust the host on which your daemon runs; i.e. taking into account that your "build" stage is preserved as an image; someone with access to that image would be able to get access to your secret.

thaJeztah on 11 Nov 2018

👍6

build time secrets are now possible when using buildkit as builder; see the blog post here https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

and the documentation; https://docs.docker.com/develop/develop-images/build_enhancements/

the RUN --mount option used for secrets will graduate to the default (stable) Dockerfile syntax soon

thaJeztah on 17 Sep 2020

❤4 🎉2

Thank you @thaJeztah I did just a little more digging and found that article shortly after posting (previous post is now deleted). Thanks again!

pdemro on 17 Sep 2020

❤3 🎉2 👍2

Cool. That closes the build time secrets question. Anything for runtime/devtime (ssh in OS X)?

Vanuan on 18 Sep 2020

Moby: Secrets: write-up best practices, do's and don'ts, roadmap

Features / hacks that are (mis)used for secrets

So, what's needed?

Most helpful comment

All 203 comments

1 and #2 above are generally the most common because the majority of applications support this mechanism. #3 is probably more ideal as it leaves the least "crumbs", but the application has to be developed specifically for this and often still has to have a credential to get the secret.

36443 was an attempt to expand it to a build-arg named like `SECRET` so that we can encourage users to use it as a simple work-around to the secret management problem.

Related issues

Moby: Secrets: write-up best practices, do's and don'ts, roadmap

Features / hacks that are (mis)used for secrets

So, what's needed?

Most helpful comment

All 203 comments

1 and #2 above are generally the most common because the majority of applications support this mechanism. #3 is probably more ideal as it leaves the least "crumbs", but the application has to be developed specifically for this and often still has to have a credential to get the secret.

36443 was an attempt to expand it to a build-arg named like SECRET so that we can encourage users to use it as a simple work-around to the secret management problem.

Related issues

36443 was an attempt to expand it to a build-arg named like `SECRET` so that we can encourage users to use it as a simple work-around to the secret management problem.