Docker-py: Add utility to create a build context from a git repository.

[apeace]

The reason is because it won't work if you pass a private git repository as the build URL.

See docker/compose#2856 for more information, including a simple test-case by @zkf.

Instead, docker-py should follow the same steps as docker: if the build URL is a git repository, it should clone the repo locally and pass as a tarball to the daemon.

I believe the problem is caused by how docker-py performs a build (here) instead of following the same steps as docker.

Please let me know if I'm misunderstanding the problem or if there's any other way I can help.

Here is my version information:

OS X El Capitan 10.11.2 (15C50)
docker-py==1.8.0.dev0
Python 2.7.9
Client:
 Version:      1.11.0-dev
 API version:  1.23
 Go version:   go1.6
 Git commit:   bc730f3-unsupported
 Built:        Tue Mar  8 19:01:29 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0-dev
 API version:  1.23
 Go version:   go1.6
 Git commit:   bc730f3-unsupported
 Built:        Tue Mar  8 19:01:29 2016
 OS/Arch:      linux/amd64

[shin-]

I think both options should exist. For public repositories, just passing the git URL is definitely faster.

Feel free to submit a patch to add that feature.

[apeace]

Thanks @shin-

I can work on a patch for this, though I'm sure it will take me some time :) I have a couple questions for you:

1. Are you sure this wouldn't be considered a bug? I believe (though have not yet tested) that this would also not work for public repositories if you use SSH syntax ([email protected]:user/repo.git). Essentially I think git SSH URLs will not work. Perhaps the bug should be clarified to say "git SSH URLs" instead of any "git URL"?
2. If indeed this is not a bug, could you give me some guidance on how a docker-py client might enable this feature? For example, maybe a client like docker-compose might pass a flag (e.g. to this call)? Any guidance on that would be appreciated--I will incorporate it into my patch.

[shin-]

Ah, I just tested it and it seems you are correct - public repos seem to fail too here. However, the engine definitely recognizes the syntax since it's trying to process it (Cloning into '/var/lib/docker/tmp/docker-build-git307830553'...) - so I would argue that this is actually a bug at the engine level. I'm certain it used to work before as well.
We should open an issue on the docker/docker issue tracker to report it and see if this can be fixed.

As far as 2 is concerned, the way I'd implement it is have a function in utils to clone a git repository (maybe this should be in its own package - it's actually nontrivial) and tar it into a file object (pointing to a temp file on the disk ideally). Then this can be passed to the Client.build method through the fileobj param, with custom_context set to True. If those utils exist compose can easily leverage them afterwards.

[apeace]

@shin- I'm not sure there is a bug at the engine level. If you read my last comment on docker/compose#2856 I think the problem is that the git clone is happening from inside the docker daemon. This means it doesn't have access to your SSH config.

One possible fix is docker/docker#6396, but I believe the docker binary has a simpler workaround that docker-py could use too.

The docker binary itself will perform the clone _locally_ first (when you run docker build), before sending a build context to the docker daemon. What I'm suggesting is that docker-py should emulate this same behavior. Does that seem right to you?

Therefore I am thinking the implementation in docker-py might look like this:

• Create a util to clone a git repo and tar it like you said.
• In BuildApiMixin.build, IF path begins with git@, then _always_ clone/tar the repo locally, and provide it do the docker daemon.

This way clients (like docker-compose) don't have to change, and they get a fixed behavior that should always be breaking right now.

[shin-]

My daemon is on the same machine as my client, so it should have access to my SSH config just fine, but I'm still experiencing the issue.

[dnephin]

I think the daemon probably runs as a different user, so it won't be looking in the right directory for the ssh config?

[apeace]

I believe @dnephin is right, the daemon is most likely running as a different user. I think that is why docker has the behavior of cloning locally instead of passing the remote param to the daemon.

[shin-]

Ah, good point. After copying my .ssh folder into /root, I am able to build from a git-ssh URL.

In [1]: import docker

In [2]: c = docker.Client()

In [3]: strm = c.build('[email protected]:docker/docker-py.git', stream=True)

In [4]: for line in strm:
   ...:     print line
   ...:     
{"stream":"Step 1 : FROM python:2.7\n"}

{"stream":" ---\u003e 31093b2dabe2\n"}
[...]
{"stream":"Successfully built 16580dbca2de\n"}

[shin-]

Regardless, I still think the second part of https://github.com/docker/docker-py/issues/980#issuecomment-193987937 is the best approach.

[apeace]

@shin- sure. I should have time to work on a patch over the next couple weeks.

Just to be clear, you're saying clients should have the choice to either:

• Use docker.util.clone_repo (or something), and then pass with fileobj and custom_context
• OR, pass path=git@github... and expect it will be passed to the docker daemon

Thank you again! :)

[shin-]

Just to be clear, you're saying clients should have the choice to either:

• Use docker.util.clone_repo (or something), and then pass with fileobj and custom_context
• OR, pass path=git@github... and expect it will be passed to the docker daemon

Yep! Thanks for offering to take it on!

[somombo]

Hi @apeace I recently ran into issues noted in #2856. How is the progress going? Thanks so much for taking this on!

[somombo]

@shin- I think this issue should actually be labelled as a bug not a feature-request. Since

The docker-py documentation under client API build section indicatest that path can be a local path (to a directory containing a Dockerfile) or a remote URL.
In addition, the official compose file spec under the context section notes that, the context can either be:
" Either a path to a directory containing a Dockerfile, OR a url to a git repository."

The intent seems clear to me: There is neither any mention in that reference that using ssh git urls is an exceptional situation, nor that it an "upcoming feature". Thus from reading those references alone users should expect regular git urls to "just work". And if that expectation is not being met, then what we have is bug, not a missing feature.

My hope is that if this is escalated to being the bug that it is, it will be quashed more quickly. The last post on this thread was over half a year ago.

[somombo]

@apeace I agree that you should clarify the title of this thread as "SSH git URLs" instead of just "git URL", because https urls seems to be working fine.

[shin-]

This will continue to be a feature-request - docker-py passes on any remote URL it gets to the engine which knows how to process them. In some cases, missing configuration on the engine's host means this process will fail, but this is not this library's responsibility.

I still welcome pull requests that would implement this additional feature, or people can use any of the workaround suggested in this thread.

[apeace]

@somombo Apologies, I never ended up having the time to get to this--shouldn't have promised I would have! I still think it would be a great feature to have, but I'm no longer actively using docker-compose so I'm not sure I will ever find the time now :(

[shin-]

Closing, see https://github.com/docker/docker-py/pull/1508#issuecomment-351596768 for rationale.

[kwidholm-tm]

The intent seems clear to me: There is neither any mention in that reference that using ssh git urls is an exceptional situation, nor that it an "upcoming feature". Thus from reading those references alone users should expect regular git urls to "just work". And if that expectation is not being met, then what we have is bug, not a missing feature.

My hope is that if this is escalated to being the bug that it is, it will be quashed more quickly. The last post on this thread was over half a year ago.

Just to say that I followed the threads to this thread due to having read the documentation and having run into this exact issue

$ docker-compose up
Building https-proxy
ERROR: error fetching: fatal: cannot run ssh: No such file or directory
fatal: unable to fork
: exit status 128

Documentation states this should work:

build: [email protected]:ORG_NAME/REPO.git

It does not.

The workarounds (github tokens + https) 1) only work for certain situations, and 2) still don't allow docker compose /docker py to work as documented.

So I agree with @somombo here. We can blame the engine or docker.py. It doesn't matter. In the end, the feature does not work as documented.

So perhaps the bug is in the documentation, and the documentation should be changed to reflect that building in docker compose via a private github repo over ssh is _not_ supported. But I would imagine I would want to write feature parity between the two engines so they are interoperable.

[natcl]

I have the same problem here:

$ docker-compose up
Building myservice
ERROR: error fetching: fatal: cannot run ssh: No such file or directory
fatal: unable to fork
: exit status 128

docker build in the cli works with git@bitbucket...
but fails with the above error when in a docker-compose file.

[renevall]

I was hoping I could use this instead of git submodules or git tree. Too bad it is not supported.