Testcontainers-java: "No such image: testcontainers/ryuk:0.3.0"

[gesellix]

Testcontainers 1.15.0 on Docker Engine 20.10/Docker for Mac 2.5.4 fails with the following stacktrace:

org.testcontainers.containers.ContainerLaunchException: Container startup failed

    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:308)
    at org.testcontainers.spock.TestcontainersMethodInterceptor.startContainers_closure3(TestcontainersMethodInterceptor.groovy:83)
    at groovy.lang.Closure.call(Closure.java:405)
    at groovy.lang.Closure.call(Closure.java:421)
    at org.testcontainers.spock.TestcontainersMethodInterceptor.startContainers(TestcontainersMethodInterceptor.groovy:80)
    at org.testcontainers.spock.TestcontainersMethodInterceptor.interceptSetupSpecMethod(TestcontainersMethodInterceptor.groovy:25)
    at org.spockframework.runtime.extension.AbstractMethodInterceptor.intercept(AbstractMethodInterceptor.java:36)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:97)
    at org.spockframework.spring.SpringInterceptor.interceptSetupSpecMethod(SpringInterceptor.java:37)
    at org.spockframework.runtime.extension.AbstractMethodInterceptor.intercept(AbstractMethodInterceptor.java:36)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:97)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=couchdb:1.7.2, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
    ... 16 more
Caused by: com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:241)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:125)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:595)
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:91)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:203)
    at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
    at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
    at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
    at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
    at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:66)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1276)
    ... 18 more

[bsideup]

@gesellix do you have a mirror configured? Or is this a response from Docker Hub?

[gesellix]

I suppose that the 404 is the local Docker Engine and no lookup to any registry is made. In this case I don't use any private registry, the engine is directly connecting to the Docker Hub. Manual testcontainers/ryuk:0.3.0 fixes the issue.

[gesellix]

As far as I know the docker cli had some logic like this for docker run (pseudo code):

try {
  create _container(image)
}catch(e){
  if e.status == 404 {
    pull_image(image)
    create _container(image)
  }
}

I don't understand, though, why we now run into such an issue. The only thing I'm aware of is https://github.com/docker/cli/pull/1498, which _might_ be related.

[gesellix]

Did/does Testcontainers pull images, in this case ryuk, before trying to create containers?

[keeganwitt]

I'm hitting this today too. In my case, for a Postgres container. I tried setting ryuk.testcontainer.image=testcontainersofficial/ryuk:0.3.0. It couldn't pull that image either. They definitely exist in Docker Hub though.

The other thing I suspected was maybe we're hitting the Docker Hub pull limits? I thought since this is communicating with the daemon it should use the auth configured on the host, but possibly I'm misunderstanding.

[gesellix]

... so maybe the pull for ryuk is performed unauthenticated?

[keeganwitt]

That's what I was worried about, yea. But I'm not sure if that's what's going on or not yet.

[bsideup]

@gesellix we definitely pull the image if it is not available.

The other thing I suspected was maybe we're hitting the Docker Hub pull limits?

In that case, an error would differ (unless the Docker Hub team have decided that 404 is a perfect http status code for a rate limited response, which should be 5xx instead)

[keeganwitt]

When we got the error before, from running docker commands directly in a job, we got a message that specifically said we hit pull limits. But I don't know the HTTP status that the docker binary received in that case, so I wasn't sure if the message was possibly being hidden by TestContainers or not.

[gesellix]

While our GitHub Actions still work (same Testcontainers version, but differend Docker Engine/operating system), I guess this is mainly related to Docker for Mac. I can give it a try with an older Docker4Mac release tomorrow.

[keeganwitt]

In my case, it's passing locally on Mac with latest Docker for Mac stable (but I have those images in my cache though) and failing on GitLab.

[bsideup]

404 and {"message":"No such image: testcontainers/ryuk:0.3.0"} is what we actually get from the API.

Also, testcontainers/* images are exempt from rate limiting, or at least that's what they told us :)

[bsideup]

oh wait, I think I know what is it...

[keeganwitt]

Well, that rules out that possibility then, at least. Maybe Docker Hub is having some problem? I just tried disabling ryuk and then it said 404 with No such image: alpine:3.5

[gesellix]

For me Docker Hub seemed to be ok, docker pull *ryuk made it work for me... well... maybe the other images have been in the local cache 🤔

[keeganwitt]

Good point. docker pull didn't break for me either locally.

[gesellix]

https://status.docker.com/pages/533c6539221ae15e3f000031 looks ok

[bsideup]

Ok, "filter by image name" query parameter in /images/json got removed, and now this condition fails:
https://github.com/testcontainers/testcontainers-java/blob/d135a2605401f6c663aab4e7edc6d6d76716f930/core/src/main/java/org/testcontainers/DockerClientFactory.java#L330

I just submitted #3575 with a fix, will be included in 1.15.1

[keeganwitt]

I suppose I should mention too that I had tried with 1.15.0-rc2 and 1.15.0.

[keeganwitt]

Ok, "filter by image name" query parameter in /images/json got removed, and now this condition fails:

https://github.com/testcontainers/testcontainers-java/blob/d135a2605401f6c663aab4e7edc6d6d76716f930/core/src/main/java/org/testcontainers/DockerClientFactory.java#L330

I just submitted #3575 with a fix, will be included in 1.15.1

Ah, so a change in Docker Hub API?

I deleted ryuk image locally and ran test again, oddly passed again.

[bsideup]

@keeganwitt Docker's API. Although the query param was deprecated (I wish we could run Docker in a strict API mode - will explore)

Sorry for this. We will release a hotfix ASAP. Meanwhile, consider pre-pulling testcontainers/ryuk:0.3.0 and alpine:3.5 :(

[keeganwitt]

@keeganwitt Docker's API. Although the query param was deprecated (I wish we could run Docker in a strict API mode - will explore)

Sorry for this. We will release a hotfix ASAP. Meanwhile, consider pre-pulling testcontainers/ryuk:0.3.0 and alpine:3.5 :(

Yea, I'd thought of that, but I'm not sure it's possible with GitLab's Docker Executor. It should be possible to run it as a script with shell executor instead though I suppose. I'm still confused why it worked locally after deleting the ryuk image though... Maybe different Docker daemon versions?

[gesellix]

Thanks @bsideup for the quick fix!

[DaspawnW]

Are you planning to backport this to work also with junit 4?

[bsideup]

@DaspawnW this is not junit specific and, once released, will work with any type of integration (junit4, junit jupiter, spock, manual container lifecycle)

[arhohuttunen]

@keeganwitt did you ever find a reasonable workaround for builds running in Gitlab? We have been looking at this for a day now without much success. It works if you manually pre-pull the images, but we are using docker-machine to autoscale the runners in EC2, so manual work is not really an option.

[jdelucaa]

@bsideup I am seeing this also in 1.14.0.

[bsideup]

@jdelucaa yes, this Docker API change applies to most of Testcontainers versions.

[keeganwitt]

@keeganwitt did you ever find a reasonable workaround for builds running in Gitlab? We have been looking at this for a day now without much success. It works if you manually pre-pull the images, but we are using docker-machine to autoscale the runners in EC2, so manual work is not really an option.

Not really. I have exactly the same setup. For now, we just commented the tests out, since a fix is forthcoming. A few ideas came to mind, but I haven't really thought through them yet.

1. Customize the AMI to bake in the images. Not totally sure if that'd work, and seems like a bad idea.
2. Put the docker pull commands as a user data script.
3. Don't use the Docker Executor, and instead use the Shell Executor and first pull the images. Assuming that works, you'd have to specify the executor on the runner itself, so you'd need a separate set of runners to do this (which is possible, just give those runners a different tag).
4. Use DinD (Docker in Docker), and execute your build in a Dockerfile, rather than a script that runs on an image. Might take some fiddling to get the volumes mounted correctly, I've never tried to do that before.

None of these seemed great. If one of them sounds promising, I can explain in a little more detail what I was thinking, though there may be gotchas I haven't thought of. Offhand, the user data script seems like the most promising to me.

[arhohuttunen]

We thought about option 1, but quickly discarded that idea. We also tried option 2 but apparently Docker isn’t installed at that point yet, so didn’t really proceed further with that. Neither 3 or 4 felt like a good idea, so I guess we’ll just skip the tests using testcontainers for now.

Thanks for sharing, though.

[keeganwitt]

@arhohuttunen I'm now thinking this broke because we upgraded GL Runners this week, which upgraded Docker version. So downgrading should fix that. Unless others didn't upgrade and still ran into this? I could have sworn we had tests pass after the upgrade, but I'm not sure what else could have changed.

I upgraded Docker for Mac to 3.0 (which has Docker 20.10 in it) this morning, and the tests now fail locally too.

[arhohuttunen]

@keeganwitt I think on the runners we are using the docker stable tag, which should still point to 19.03.14 according to this:
https://hub.docker.com/layers/docker/library/docker/stable/images/sha256-8f71deccd0856d8a36db659a8c82894be97546b47c1817de27d5ee7eea860162?context=explore

[keeganwitt]

@arhohuttunen Sorry, I didn't mean the runners image, I meant the machines on which the runner image runs (where the daemon lives). We use https://github.com/npalm/terraform-aws-gitlab-runner, which would upgrade that. This applies to private runners, not the shared ones that GL manages. I dunno what schedule those are upgraded on, we don't use them.

I've been talking with another of our engineers and he said userdata isn't the same as ec2-userdata. I didn't realize there were 2.

[Urokhtor]

We use a hard-coded AMI in our runners and haven't changed that lately, so that should not be the root cause in our case.

[keeganwitt]

Actually, some builds passed after our upgrade too, so it shouldn't have been that. I'm confused why yesterday was the breaking day.

[Urokhtor]

I'll take previous comment back because those runners install Docker from official repo which does serve 20.10.

[jeantil]

I'm encountering this issue on apaceh CI builds for the apache/james project (https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/16/pipeline)
I tried pulling the image explicitly before running the tests but it still fails. we are very much looking forward to the 1.15.1 hotfix

edit
I was misled by a comment above that referred to testcontainersofficial instead of testcontainers I assume this was some kind of custom setup. The tests try to get testcontainers/ryuk:0.3.0 by default no testcontainersofficial/ryuk:0.3.0

[keeganwitt]

@jeantil That's correct, testcontainers is the default, testcontainersofficial was just a misguided thing I attempted early on (overriding the image in testcontainers properties). I saw testcontainersofficial mentioned in an issue where they were discussing Docker Hub and Quay. Sorry for the confusion.

[mderouet]

When can we expect to have this 1.15.1 release ? (In order to know whether we need to find a workaround for this or just wait for it)

[bsideup]

@mderouet the release is expected for later today (tsss ;))

[bsideup]

released in 1.15.1 🎉

[telapo]

It works, thanks

[alex-sky-cloud]

I got an update from Docker today, and I got Docker desktop-3.0.1 (50773).

Now I get an error while running Test container.

...... <<< ERROR! org.testcontainers.containers.ContainerLaunchException: Container startup failed Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:11.7, imagePullPolicy=DefaultPullPolicy()) Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: testcontainersofficial/ryuk:0.3.0"}

I assumed you fixed it. I also disabled the Use gRPC FUSE for file sharing option, but it didn't help.

How can I fix this ?

[bsideup]

@alex-sky-cloud this is another issue, unrelated to the file sharing, and it is fixed in 1.15.1, please update.

[alex-sky-cloud]

I'm sorry. What should I update ?

[bsideup]

@alex-sky-cloud the project you're reporting to - Testcontainers :D

[alex-sky-cloud]

I have the latest version of docker.

Which project should I update ?

I executed the

docker system prune-af

command and only after that the error disappeared.

So, is this how you will need to do it every time ?

[bsideup]

@alex-sky-cloud No. Just use the latest (1.15.1) version of Testcontainers.

[davoutuk]

This issue is still live for me when I tried using this on a Spring Boot/Postgres app. The fix of separating downloading the 'testcontainer' docker image works.
I would suggest adding a note to the docs to explain that this is a pre requiste

[bsideup]

@davoutuk with Testcontainers 1.15.1? There was a bug that got fixed in 1.15.1, there isn't such prerequisite

[AlexeyPashchenko]

just pull it manualy by console "docker pull testcontainers/ryuk:0.3.0"

[hafizali05]

i understand this might not work for every case but what fixed for me was to clean purge data and reset to factory settings

is this the best way? i dont know as i am new to docker and test containers.

[bsideup]

@hafizali05 this is not usually needed, unless your images cache is corrupted.

[hafizali05]

@hafizali05 this is not usually needed, unless your images cache is corrupted.

i cant reproduce this bit again so if this persist again i might try to find out a way to only clear image cache. this ofcourse removed all my images and containers. @bsideup

[jeantil]

I confirm that after upgrading to test containers 1.15.1 we no longer have this issue on apache/james-project

[jeffnyauke]

I confirm that after upgrading to test containers 1.15.1 I no longer have this issue

[PauloPortfolio]

In my case, I have solved this problem cleaning all my docker-images.

However, upgrading, my POM.XML, to test containers 1.15.1, already, is a solution to his trouble.

[MiguelAngel82]

Hi. It seems that this error is happening again.
With the following versions:

• Docker Desktop for Mac 3.3.1.
• Docker engine 20.10.5
• Testcontiners 1.15.3
• Gradle 5.4

It's failing with the same error:

java.lang.RuntimeException: Could not create new instance of class org.jboss.arquillian.test.impl.EventTestRunnerAdaptor
    at org.jboss.arquillian.test.spi.SecurityActions.newInstance(SecurityActions.java:146)
    at org.jboss.arquillian.test.spi.SecurityActions.newInstance(SecurityActions.java:89)
    at org.jboss.arquillian.test.spi.TestRunnerAdaptorBuilder.build(TestRunnerAdaptorBuilder.java:49)
    at org.jboss.arquillian.junit.AdaptorManager.initializeAdaptor(AdaptorManager.java:21)
    at org.jboss.arquillian.junit.ArquillianTestClassLifecycleManager.beforeTestClassPhase(ArquillianTestClassLifecycleManager.java:24)
    at org.jboss.arquillian.junit.ArquillianTestClass$1.evaluate(ArquillianTestClass.java:39)
    at org.spockframework.runtime.extension.builtin.ClassRuleInterceptor.intercept(ClassRuleInterceptor.java:39)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:97)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
    at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
    at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
    at org.jboss.arquillian.test.spi.SecurityActions.newInstance(SecurityActions.java:144)
    ... 28 more
Caused by: com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:241)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:125)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:595)
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:91)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:203)
    at com.adaptavist.testfixtures.testcontainers.TestHostApplicationContainerProvider.<init>(TestHostApplicationContainerProvider.groovy:23)
    at com.adaptavist.testfixtures.application.BaseTestHostApplication.<init>(BaseTestHostApplication.groovy:18)
    at com.adaptavist.testfixtures.application.JiraCoreTestHostApplication.<init>(JiraCoreTestHostApplication.groovy:9)
    at com.adaptavist.arquillian.TestHostApplicationInstanceRegistrar.registerScriptRunnerFixture(TestHostApplicationInstanceRegistrar.groovy:20)
    at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
    at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
    at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
    at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
    at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
    at org.jboss.arquillian.core.impl.ManagerImpl.start(ManagerImpl.java:253)
    at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.<init>(EventTestRunnerAdaptor.java:61)
    ... 29 more

I've tried to downgrade Docker Desktop and it was failing until Docker Desktop Community 2.5.0.1, which is using Docker engine 19.03.13. So with this last version (and keeping the same Testcontainers versions) is working.

Also, I've applied the workaround of manually downloading the image testcontainers/ryuk:0.3.0 and with this workarodund is working for the last Docker Desktop version (3.3.1).

So please, if someone else is facing the same problem, maybe it's worth to have a look if something has broken with this last Docker version.

Thank you!

[bsideup]

@MiguelAngel82 Testcontainers 1.15.3 depens on Ryuk 0.3.1, meaning that you're still using an older version of Testcontainers

[MiguelAngel82]

@bsideup Thank you for your response! Sorry, it was my fault. Thanks to your response I've dug deeper into the project and I've realised that we were using another Testcontainers dependency, org.testcontainers:selenium that was not updated.
So now, both dependencies has been updated to the latest version (1.15.3) and it's properly working ;)

Thank you and I apologise for the inconvenience.