Compose: Not able to use private git repo for build context in Docker Compose 1.6.2

[ripperdoc]

I have a private repo with src and Dockerfile to build from, and I am connecting to a remote docker host using docker-machine. I am able to run the following:

docker build -t MYCONTAINER [email protected]:myrepo/myrepo.git#master

However, if I try to run docker-compose, with the following build command:

  MYCONTAINER:
    build: [email protected]:MYREPO/myrepo.git#master

It fails due to github authentication, although I believed that the git cloning happens locally in the same way as docker build? Here is the verbose output form docker-compose:

compose.config.config.find: Using configuration files: ./production-dc.yml
docker.auth.auth.load_config: File doesn't exist
compose.cli.command.get_client: docker-compose version 1.6.2, build 4d72027
docker-py version: 1.7.2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
compose.cli.command.get_client: Docker base_url: https://REMOTEIP:2376
compose.cli.command.get_client: Docker version: KernelVersion=3.13.0-58-generic, Os=linux, BuildTime=2016-02-22T21:37:01.910365059+00:00, ApiVersion=1.22, Version=1.10.2, GitCommit=c3959b1, Arch=amd64, GoVersion=go1.5.3
compose.service.build: Building MYCONTAINER
compose.cli.verbose_proxy.proxy_callable: docker build <- (pull=False, stream=True, nocache=True, tag=u'containers_fablr', buildargs=None, rm=True, forcerm=False, path='[email protected]:MYREPO/myrepo.git#master', dockerfile=None)
docker.api.build._set_auth_headers: Looking for auth config
docker.api.build._set_auth_headers: No auth config in memory - loading from filesystem
docker.auth.auth.load_config: File doesn't exist
docker.api.build._set_auth_headers: No auth config found
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object _stream_helper at 0x104d01d70>
ERROR: compose.cli.main.log_api_error: Error trying to use git: exit status 128 (Cloning into '/var/lib/docker/tmp/docker-build-git575327272'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
)

I replaced my URLs and repo names, as you can see. Not sure what the auth_headers errors relate to.

[jaycyb]

+1

compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=dockerteamcity', u'com.docker.compose.service=server', u'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.service.execute_convergence_plan: dockerteamcity_mysql_1 is up-to-date
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- (u'dockerteamcity_server')
compose.service.build: Building server
compose.cli.verbose_proxy.proxy_callable: docker build <- (pull=False, nocache=False, stream=True, tag=u'dockerteamcity_server', buildargs=None, forcerm=False, rm=True, path='/home/jbcebe/project/build/docker-teamcity', dockerfile='ServerDockerFile')
docker.api.build._set_auth_headers: Looking for auth config
docker.api.build._set_auth_headers: No auth config in memory - loading from filesystem
docker.auth.auth.load_config: File doesn't exist
docker.api.build._set_auth_headers: No auth config found

[jmahowald]

+1 My assumption is that unlike docker-build the pull is coming from the docker-machine, which doesn't have the correct credentials (or in my case, the correct CA chain to the server)

[gittycat]

I believe I'm hitting the same issue but not at build time.

My docker-compose.yml references an image quay.io/mybiz/postgres and I'm now getting this error while doing a docker-compose up -d.

ERROR: Error: Status 403 trying to pull repository mybiz/postgres: "{\"error\": \"Permission Denied\"}"

I can however download the image successfully using docker pull quay.io/mybiz/postgres.

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 21:49:11 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      swarm/1.1.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   7e9c6bd
 Built:        Wed Mar  2 00:

docker-compose version 1.6.2, build 4d72027
docker-py version: 1.7.2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014

[ripperdoc]

So I can report a step forward on this one. It's possible to access a private repo through the url: https://username:[email protected]/username/repo.git instead of the ssh type url ([email protected]:..) . Additionally, instead of password in cleartext you can generate a token in Github and use that instead. And this works with docker-compose!

As @jmahowald stated, the build command is run from within the docker environment in docker-compose (e.g. on the VM, which lacks my SSH credentials), whereas in docker build it is run locally on my Mac (which has the credentials). However, I still consider this a bug as the behaviour is different between docker build and docker-compose build.

[somombo]

Hi all, More information on what's causing this issue is available at

docker/docker-py#980

[marcellodesales]

Based on @ripperdoc's insight, we can use a Github Personal Token that has read access to the github repo. Use the token as the username with an empty password:

https://${TOKEN}:@github.company.com/org/repo.git

Here's an example:

services:
  reference:
    build: "https://f4d24eed3909ad8766e29fd2500e80cb33d153a8:@github.company.com/services-configuration/spring-cloud-config-reference-service.git"
    command: ["./wait-for-it.sh", "--timeout=60", "config-server:8888", "--", "java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

This works! At this point, you can define an environment variable for the token such your OPS team can provide their own token.

services:
  reference:
    build: "https://${TOKEN:-f4d24eed3909ad8766e29fd2500e80cb33d153a8}:@github.company.com/services-configuration/spring-cloud-config-reference-service.git"
    command: ["./wait-for-it.sh", "--timeout=60", "config-server:8888", "--", "java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

Finally, you can specify which branch to use with a suffix #branch in the URL. The example below forces the client to clone from the develop branch.

services:
  reference:
    build: "https://${TOKEN:-f4d24eed3909ad8766e29fd2500e80cb33d153a8}:@github.company.com/services-configuration/spring-cloud-config-reference-service.git#develop"
    command: ["./wait-for-it.sh", "--timeout=60", "config-server:8888", "--", "java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

[chribsen]

If someone is using Bitbucket the equivalent to Personal access tokens are _App passwords_. This can be found under _Bitbucket Settings_ --> _Access Management_ --> _App passwords_

[dan-kez]

I would love to see this work out of the box without using github tokens for auth. It's unfortunate to have everyone create personal tokens to pull down a private repo image.

[kwidholm-tm]

Creating personal tokens won't work in certain contexts of distributed workflows with private repos. Documentation states building from git over ssh should work in docker compose and it does not.

[fcce]

for gitlab "https://oauth2:${TOKEN}@gitlab.com/xxx/xxx.git" works

[MBeijer]

Any update on this? Any of the suggested ways to do it doesn't really work on a dedicated private bitbucket which runs on port 7999. I've tried personal access tokens without luck.
Please fix this!

Tried on both MacOS Mojave and Windows 10

[lrushl]

Is there a way to use azure devops private git repo for build context in Docker Compose?

I'm getting the following error
ERROR: error downloading remote context https://[email protected]/MY-ORG/PROJECTNAME/_git/reponame: failed to GET https://[email protected]/MY-ORG/PROJECTNAME/_git/reponame with status 401 Unauthorized:

docker build https://[email protected]/MY-ORG/PROJECTNAME/_git/reponame
resuls with same error
failed to resolve httpcontext: invalid response status 401

I'm using windows 10. Correct credentials are set in the windows credentials store so it is possible to pull changes in cmd with git.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.

[bdecarne]

too bad this issue was closed :( store token & credentials in docker-compose is insane !

[alphachai]

Is there any possibility of this issue being reopened? It would be ideal if docker-compose used the current user's git / ssh credentials.

[nerdkid93]

Can this issue get reopened please? It's rather jarring for docker build git@gitserver:org/repo.git#branch to work as expected when docker-compose build service does work when this is the docker-compose.yaml file:

version: '3.8'
services:
  service:
    build: git@gitserver:org/repo.git#branch

I believe docker-compose build should have feature parity with the normal docker build CLI

[dAnjou]

Seems to be the same as https://github.com/docker/compose/issues/2856.