Compose: docker-compose cannot use gcloud credential helper on Linux (pyinstaller release)

[xmartinez]

Description of the issue

docker-compose pull fails to retrieve images from Google Cloud Repository (using gcloud credential helper), with the following exception:

Pulling hello-world ... 
ERROR: gcloud failed to load: 'module' object has no attribute 'HTTPSConnection'
    gcloud_main = _import_gcloud_main()
    import googlecloudsdk.gcloud_main
    from googlecloudsdk.calliope import base
    from googlecloudsdk.calliope import display
    from googlecloudsdk.calliope import display_taps
    from googlecloudsdk.core.resource import resource_printer_base
    from googlecloudsdk.core.resource import resource_projector
    from apitools.base.py import encoding as protorpc_encoding
    from apitools.base.py.base_api import *
    from apitools.base.py import http_wrapper
    import httplib2
    from httplib2.python2.httplib2 import *
    class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):

This usually indicates corruption in your gcloud installation or problems with your Python interpreter.

Please verify that the following is the path to a working Python 2.7 executable:
    /usr/bin/python2

If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 executable.

If you are still experiencing problems, please reinstall the Cloud SDK using the instructions here:
    https://cloud.google.com/sdk/

ERROR: for hello-world  Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".',)
Traceback (most recent call last):
  File "site-packages/dockerpycreds/store.py", line 71, in _execute
  File "subprocess.py", line 336, in check_output
  File "subprocess.py", line 418, in run
subprocess.CalledProcessError: Command '['/usr/bin/docker-credential-gcloud', 'get']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/docker/auth.py", line 126, in _resolve_authconfig_credstore
  File "site-packages/dockerpycreds/store.py", line 33, in get
  File "site-packages/dockerpycreds/store.py", line 84, in _execute
dockerpycreds.errors.StoreError: Credentials store docker-credential-gcloud exited with "".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 729, in pull
  File "compose/project.py", line 556, in pull
  File "compose/parallel.py", line 105, in parallel_execute
  File "compose/parallel.py", line 203, in producer
  File "compose/project.py", line 549, in pull_service
  File "compose/service.py", line 1163, in pull
  File "site-packages/docker/api/image.py", line 380, in pull
  File "site-packages/docker/auth.py", line 47, in get_config_header
  File "site-packages/docker/auth.py", line 94, in resolve_authconfig
  File "site-packages/docker/auth.py", line 143, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".',)
[9076] Failed to execute script docker-compose

Context information (for bug reports)

$ docker-compose --version
docker-compose version 1.21.1, build 5a3f1a3Output of "docker-compose version"

$ docker --version
Docker version 18.03.0-ce, build 0520e24

$ docker-compose config
services:
  hello-world:
    image: gcr.io/library/hello-world
version: '3.0'

Steps to reproduce the issue

After installing the Linux pyinstaller release:

wget -O $HOME/.local/bin/docker-compose https://github.com/docker/compose/releases/download/1.21.1/docker-compose-Linux-x86_64
chmod +x $HOME/.local/bin/docker-compose

and setting up gcloud credential helper in ~/.docker/config.json:

{
  "credHelpers": {
    "gcr.io": "gcloud"
  }
}

Trying to pull images (docker-compose pull) for the following sample docker-compose.yaml:

version: "3"
services:
  hello-world:
    image: gcr.io/library/hello-world

fails with the above traceback.

Additional information

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.10
Release:    17.10
Codename:   artful

$ gcloud --version
Google Cloud SDK 200.0.0
alpha 2018.04.30
beta 2018.04.30
bq 2.0.33
core 2018.04.30
gsutil 4.31

[xmartinez]

As a workaround, installing using pip (https://docs.docker.com/compose/install/#install-using-pip) works correctly.

[xmartinez]

After digging up a little bit, it seems that the pyinstaller-based release messes up LD_LIBRARY_PATH. The gcloud credential helper is run as a subprocess by docker-compose and inherits the LD_LIBRARY_PATH value set by pyinstaller.

As gcloud runs with Python 2, it gets Python 2 stdlib .py modules, but the pyinstaller LD_LIBRARY_PATH value causes it to load Python 3 stdlib .so modules (included with docker-compose), so hilarity ensues.

This is a documented issue of pyinstaller packaged applications that fork-exec other apps: https://pyinstaller.readthedocs.io/en/v3.3.1/runtime-information.html#ld-library-path-libpath-considerations.

Unfortunately, the subprocess call for the credential helper is two levels deep in docker-compose dependency tree: the credential helper is forked by dockerpycreds (https://github.com/shin-/dockerpy-creds/blob/master/dockerpycreds/store.py#L70), which is called by docker Python api, which is called by docker-compose).

Thus, modifying the subprocess.check_output() call as suggested by pyinstaller docs is likely out of the question.

As a (somewhat dirty) fix, docker-compose could monkey-patch subprocess.Popen (as suggested in this comment: https://github.com/pyinstaller/pyinstaller/issues/1759#issuecomment-177996408), whenever it is running as part of a pyinstaller release.

[shin-]

Thanks for the report!

We can probably update docker-pycreds to optionally take some environment overrides, which would let us implement the workaround suggested in the pyinstaller docs.

[DonkeyOatie]

I am seeing this same problem on OS X with docker-compose build

gcloud --version
Google Cloud SDK 199.0.0
bq 2.0.33
core 2018.04.20
gsutil 4.30

docker-compose --version
docker-compose version 1.21.1, build 5a3f1a3

docker --version
Docker version 18.03.1-ce, build 9ee9f40

Building site
Traceback (most recent call last):
  File "/Users/nathan/code/gcloud-sdk/lib/gcloud.py", line 88, in <module>
    main()
  File "/Users/nathan/code/gcloud-sdk/lib/gcloud.py", line 64, in main
    import traceback
ImportError: No module named traceback
Traceback (most recent call last):
  File "site-packages/dockerpycreds/store.py", line 71, in _execute
  File "subprocess.py", line 336, in check_output
  File "subprocess.py", line 418, in run
subprocess.CalledProcessError: Command '['/Users/nathan/code/gcloud-sdk/bin/docker-credential-gcloud', 'get']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/docker/auth.py", line 126, in _resolve_authconfig_credstore
  File "site-packages/dockerpycreds/store.py", line 33, in get
  File "site-packages/dockerpycreds/store.py", line 84, in _execute
dockerpycreds.errors.StoreError: Credentials store docker-credential-gcloud exited with "".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 282, in build
  File "compose/project.py", line 373, in build
  File "compose/service.py", line 1047, in build
  File "site-packages/docker/api/build.py", line 251, in build
  File "site-packages/docker/api/build.py", line 305, in _set_auth_headers
  File "site-packages/docker/auth.py", line 94, in resolve_authconfig
  File "site-packages/docker/auth.py", line 143, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".',)
[85092] Failed to execute script docker-compose

[sebglon]

Always the same for me

docker-compose --version
docker-compose version 1.21.2, build a133471

docker --version
Docker version 18.03.1-ce, build 9ee9f40

[shin-]

@nkhumphreys @xmartinez @sebglon Can you check that #6012 resolves the problem? I posted links to the binary artifacts you can use to verify.

Thanks!

[mikzuit]

Hey @shin- I think this issue in any way should point to https://github.com/GoogleCloudPlatform/github-actions/issues/128#issuecomment-658278079 that fix it. My OS: Ubuntu 20.04, docker-compose 1.26.2, I was having Google Cloud SDK 301.0.0 and have to downgrade it to 297.0.1-0. Could you please check this out?

Update for New People with this issue:
Also if you want to have the latest 3.0.7 gcloud installed here is the workaround to fix the open issue. hope this helps to latest people to comment ( @tutunana @paulcdejean ) . This is just a reminder this issue is closed because of that, that is been pointed before in the thread :
https://github.com/GoogleCloudPlatform/github-actions/issues/128#issuecomment-680316388

[tutunana]

I have same issue and have to use pip install docker-compose as @xmartinez did above

[paulcdejean]

Hello I'm having this issue after a recent update to the google cloud SDK.

Before I wasn't having this issue but after I ran:

sudo apt-get update && sudo apt-get --only-upgrade install google-cloud-sdk-app-engine-python-extras google-cloud-sdk-anthos-auth google-cloud-sdk-app-engine-go google-cloud-sdk-minikube kubectl google-cloud-sdk-kind google-cloud-sdk-app-engine-java google-cloud-sdk-firestore-emulator google-cloud-sdk google-cloud-sdk-datastore-emulator google-cloud-sdk-kpt google-cloud-sdk-bigtable-emulator google-cloud-sdk-datalab google-cloud-sdk-cbt google-cloud-sdk-spanner-emulator google-cloud-sdk-app-engine-grpc google-cloud-sdk-app-engine-python google-cloud-sdk-cloud-build-local google-cloud-sdk-pubsub-emulator google-cloud-sdk-skaffold

In order to update my cloud SDK, I'm having this issue.

[paulcdejean]

There's a bug in google cloud SDK version 301, which was released on July 14th, which seems to be causing this:

https://issuetracker.google.com/issues/166482953

As a workaround, you can downgrade your cloud SDK version to version 300, for example by doing:

sudo apt-get update && sudo apt-get install google-cloud-sdk=300.0.0-0

[lewislbr]

Another option is to run export CLOUDSDK_PYTHON=python2.