Testcontainers-java: docker-credential-osxkeychain error on OS X

[mobiletoly]

Getting this error with TestContainer 1.9.1 (reproduced on 1.9.0 as well)

2018-10-15 11:18:43.097 [main] ERROR o.t.s.o.z.exec.ProcessExecutor -  - Could not start process:
java.io.IOException: Cannot run program "docker-credential-osxkeychain": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.testcontainers.shaded.org.zeroturnaround.exec.ProcessExecutor.invokeStart(ProcessExecutor.java:1016)
    at org.testcontainers.shaded.org.zeroturnaround.exec.ProcessExecutor.startInternal(ProcessExecutor.java:989)
    at org.testcontainers.shaded.org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:925)
    at org.testcontainers.utility.RegistryAuthLocator.runCredentialProgram(RegistryAuthLocator.java:323)
    at org.testcontainers.utility.RegistryAuthLocator.runCredentialProvider(RegistryAuthLocator.java:219)
    at org.testcontainers.utility.RegistryAuthLocator.authConfigUsingStore(RegistryAuthLocator.java:186)
    at org.testcontainers.utility.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.java:120)
...

when revert back to 1.8.3 version - everything starts working again and I don't see this error anymore.

[rnorth]

Hi Toly,

Hopefully we can resolve this - could you just check a few things?

1. If you look at your ~/.docker/config.json file do you see osxkeychain mentioned there as a credential store or credential helper?
2. Do you have a program named docker-credential-osxkeychain on your system anywhere? Is it on your PATH? (FWIW it is a standard part of the Docker for Mac installation so should be there somewhere)
3. Do you actually need to pull images from a private docker registry, or are you only using public images?

Thanks

Richard

[mobiletoly]

Hi Richard, thanks for such a quick reply. As I mentioned previously - it started to break only with 1.9 version and yes - I do have "docker-credential-osxkeychain" in my path (I can run it in terminal). I took a look at config.json as well:

{
  ...
  "credsStore" : "osxkeychain"
}

I use public images.

[rnorth]

That is very strange - please could you try dumping the PATH as seen by the Java process (e.g. grab the value of System.env("PATH") somewhere in your test code).

The error _implies_ that this executable is not in a place where Testcontainers can find it.

As a short term fix you could try removing the credsStore line from your docker config file; this would stop Testcontainers from trying to use it. I'm not sure if the line would automatically be reinstated later or not, though, so this isn't a long term fix.

[mobiletoly]

Interesting. So it does not look like a TestContainers problem. I tried System.getenv("PATH") and it returns null on my Mac. This is weird, have to investigate it. So what I did is added PATH="/usr/local/bin/" in IntelliJ's Run Configuration (Environment field) and it is started to work. Thank you so much for System.getenv("PATH") hint, still have no clue why it returns null on my computer but at least I'm unblocked with using TestContainers 1.9.1.

Also "./gradlew test" running in Terminal working like a charm,because it is able to pick up a correct PATH (I printed it in a console and it shows a full path), while in IntelliJ environment System.getenv("PATH") returns null.

BTW, do you have any idea why it works on 1.8.x setup?

[mobiletoly]

Seems like it works even if I don't add "PATH=/usr/local/bin" in IntelliJ's environment variable. The only difference is that if I add it, then I don't see that exception I printed in my initial post and it runs in my unit tests successfully. If I don't add that PATH, then running under IntelliJ will result in that "ProcessExecutor - - Could not start process" exception, but then it is still able to run TestContainers with my unit tests. So I guess the difference between 1.8.x and 1.9.x is that 1.9.x actually prints that exception while 1.8.x hides it?

[tapbooth]

I also had this issue with 1.9.1 and reverting to 1.8.3 suppressed the error. After some more digging, i saw in my .docker/config.json i had two entries for the cred store, but the type-case was different:
{
"credSstore" : "",
"HttpHeaders" : {
"User-Agent" : "Docker-Client/18.06.1-ce (darwin)"
},
"credsStore" : "osxkeychain"
}
After enabling and disabling the osx key chain from the docker UI (for mac) i saw that "credsStore" wasn't being used. Removing it from my config (manually) also stopped the error from showing up.

Not sure if this is a docker issue or a testcontainers issue... either way this resolved the Exception for me.

[remmeier]

I do get a very similar exception on Windows:

java.io.IOException: Cannot run program "docker-credential-": CreateProcess error=2, Das System kann die angegebene Datei nicht finden
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.testcontainers.shaded.org.zeroturnaround.exec.ProcessExecutor.invokeStart(ProcessExecutor.java:1016)
    at org.testcontainers.shaded.org.zeroturnaround.exec.ProcessExecutor.startInternal(ProcessExecutor.java:989)
    at org.testcontainers.shaded.org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:925)
    at org.testcontainers.utility.RegistryAuthLocator.runCredentialProgram(RegistryAuthLocator.java:323)
    at org.testcontainers.utility.RegistryAuthLocator.runCredentialProvider(RegistryAuthLocator.java:219)
    at org.testcontainers.utility.RegistryAuthLocator.authConfigUsingStore(RegistryAuthLocator.java:186)
    at org.testcontainers.utility.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.java:120)
    at org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig.effectiveAuthConfig(AuthDelegatingDockerClientConfig.java:43)
    at com.github.dockerjava.core.DockerClientImpl.createContainerCmd(DockerClientImpl.java:316)
    at org.testcontainers.dockerclient.AuditLoggingDockerClient.createContainerCmd(AuditLoggingDockerClient.java:32)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:255)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:237)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:235)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:220)

I do have a default Hyperv-based installation with no customizations so far. My config looks like:

{
  "credsStore": "",
  "auths": {},
  "stackOrchestrator": "swarm"
}

[rnorth]

This is interesting - I'd say that "credsStore": "" is not valid... But regardless of whether it's valid or not, it's occurring in more than one place so probably has a systemic cause.

We should guard against this.

I don't have much time right now, but maybe someone could contribute some extra validation to RegistryAuthLocator to prevent an empty string being used for a credential store / credential helper?

[MykolaMarkov]

I see the similar behavior in Eclipse on Mac. It works fine after adding PATH to environment

[nstaeger]

Same for me in Eclipse on Mac. Testcontainers 1.10.6. No Exception with version 1.7.3/1.8.3. Printing the PATH returns /usr/bin:/bin:/usr/sbin:/sbin. This happens while starting tests for a Spring Boot application.

[zhenik]

For OSX users, who is using Docker desktop:

1. make sure to uncheck Securely store Docker logins in macOS keychain
2. OR store your credentials in keychain properly.
   

1303

[rnorth]

Merged #1303 to fix this.

[cschneider]

I see the same problem.. my ~/.docker/config.json contains:
"credsStore" : "osxkeychain",
"credSstore" : "osxkeychain",

[zhenik]

@cschneider
I investigated the behaviour of docker desktop:

1. check box Securely store Docker logins in macOS keychain -> "credSstore":"osxkeychain" automatically created by docker desktop, then I checked Keychain Access, searched for docker, I was able to see newly created credentials. Run integration tests -> all ok
2. uncheck box -> "credSstore":"", docker desktop also remove docker credentials from Keychain Access. Run integration tests -> all ok
3. I manually removed whole field "credSstore":"" from .docker/config.json. Run integration tests -> all ok

I think this issue relates to docker desktop application, not testcontainers.
some screenshots:

[rnorth]

Releasing #1303 in 1.11.0 🎉

[rnorth]

@NikitaZhevnitskiy good investigation - that "credSstore" is bizarre! Maybe worth reporting this over in https://github.com/docker/for-mac ?

I'm going to close this ticket now that we're releasing a workaround, but if this problem recurs in any way from 1.11.0 onwards please open a new issue and link this one.

[bethmage]

add the directory contains the docker-credential-desktop to PATH Env.

[JasonDingJ]

I solved it by remove credsstore