Docker-java: Bitbucket Pipelines fails authorization of testcontainers to pull images due to "authorization denied by plugin pipelines: Invalid content length provided"
Hey,
I am forced to upgrade testcontainers-java to 1.15.0-rc2 due to docker-desktop for mac 2.4.0.0 causing failures. This however causes the bitbucket pipelines to fail.
The issue is also mentioned in the testcontainers-java repository.
It seems the content-length is incorrect somehow as indicated by the stack-trace line:
Caused by: com.github.dockerjava.api.exception.DockerException: Status 403: {"message":"authorization denied by plugin pipelines: Invalid content length provided"}
But do not confuse me for someone who knows what they're talking about. Anyway here's the full stack-trace for the first integration-test in the bitbucket pipeline:
INFO DockerMachineClientProviderStrategy:isApplicable - docker-machine executable was not found on PATH ([/usr/local/openjdk-11/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin])
INFO DockerClientProviderStrategy:lambda$getFirstValidStrategy$6 - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=tcp://localhost:2375
INFO DockerClientFactory:client - Docker host IP address is localhost
INFO DockerClientFactory:client - Connected to docker:
Server Version: 18.09.9
API Version: 1.39
Operating System: Alpine Linux v3.10 (containerized)
Total Memory: 31703 MB
INFO DockerClientFactory:client - Checking the system...
INFO DockerClientFactory:check - 鉁旓笌 Docker server version should be at least 1.6.0
INFO DockerClientFactory:onStart - Starting to pull image
INFO DockerClientFactory:onNext - Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)
INFO DockerClientFactory:onNext - Pulling image layers: 0 pending, 1 downloaded, 0 extracted, (1 MB/1 MB)
INFO DockerClientFactory:onNext - Pulling image layers: 0 pending, 1 downloaded, 1 extracted, (1 MB/1 MB)
ERROR SpringApplication:reportFailure - Application run failed
java.lang.ExceptionInInitializerError: null
at our.group.our.package.IntegrationTestBase$Initializer.initialize(IntegrationTestBase.kt:99) ~[test-classes/:?]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) ~[spring-boot-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) ~[spring-boot-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) ~[spring-boot-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:124) ~[spring-boot-test-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:123) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.junit.jupiter.SpringExtension.getApplicationContext(SpringExtension.java:202) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.junit.jupiter.SpringExtension.resolveParameter(SpringExtension.java:188) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameter(ExecutableInvoker.java:216) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameters(ExecutableInvoker.java:183) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:74) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeTestClassConstructor(ClassBasedTestDescriptor.java:329) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.instantiateTestClass(ClassBasedTestDescriptor.java:276) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.instantiateTestClass(ClassTestDescriptor.java:77) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.instantiateAndPostProcessTestInstance(ClassBasedTestDescriptor.java:258) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$testInstancesProvider$2(ClassBasedTestDescriptor.java:252) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at java.util.Optional.orElseGet(Optional.java:369) ~[?:?]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$testInstancesProvider$3(ClassBasedTestDescriptor.java:251) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.execution.TestInstancesProvider.getTestInstances(TestInstancesProvider.java:29) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$prepare$0(TestMethodTestDescriptor.java:106) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.prepare(TestMethodTestDescriptor.java:105) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.prepare(TestMethodTestDescriptor.java:69) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$prepare$1(NodeTestTask.java:107) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.prepare(NodeTestTask.java:107) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:75) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220) ~[junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188) ~[junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202) [junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181) [junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128) [junit-platform-launcher-1.3.1.jar:1.3.1]
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150) [surefire-junit-platform-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124) [surefire-junit-platform-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) [surefire-booter-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) [surefire-booter-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) [surefire-booter-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) [surefire-booter-2.22.2.jar:2.22.2]
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3, imagePullPolicy=DefaultPullPolicy())
at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1282) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:616) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.elasticsearch.ElasticsearchContainer.<init>(ElasticsearchContainer.java:65) ~[elasticsearch-1.15.0-rc2.jar:?]
at org.testcontainers.elasticsearch.ElasticsearchContainer.<init>(ElasticsearchContainer.java:53) ~[elasticsearch-1.15.0-rc2.jar:?]
at our.group.our.package.IntegrationTestBase$Companion$elasticContainer$1.<init>(IntegrationTestBase.kt:63) ~[test-classes/:?]
at our.group.our.package.IntegrationTestBase.<clinit>(IntegrationTestBase.kt:63) ~[test-classes/:?]
... 63 more
Caused by: com.github.dockerjava.api.exception.DockerException: Status 403: {"message":"authorization denied by plugin pipelines: Invalid content length provided"}
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:249) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:125) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:595) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.DockerClientFactory.runInsideDocker(DockerClientFactory.java:350) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:222) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:65) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:26) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1280) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:616) ~[testcontainers-1.15.0-rc2.jar:?]
at org.testcontainers.elasticsearch.ElasticsearchContainer.<init>(ElasticsearchContainer.java:65) ~[elasticsearch-1.15.0-rc2.jar:?]
at org.testcontainers.elasticsearch.ElasticsearchContainer.<init>(ElasticsearchContainer.java:53) ~[elasticsearch-1.15.0-rc2.jar:?]
at our.group.our.package.IntegrationTestBase$Companion$elasticContainer$1.<init>(AMIntegrationTestBase.kt:63) ~[test-classes/:?]
at our.group.our.package.IntegrationTestBase.<clinit>(AMIntegrationTestBase.kt:63) ~[test-classes/:?]
... 63 more
2020-10-01 12:16:50,717 main INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
and all subsequent integration tests fail with the stack trace:
ERROR SpringApplication:reportFailure - Application run failed
java.lang.NoClassDefFoundError: Could not initialize class our.group.our.package.IntegrationTestBase
at no.asta.am.core.util.AMIntegrationTestBase$Initializer.initialize(IntegrationTestBase.kt:99) ~[test-classes/:?]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) ~[spring-boot-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) ~[spring-boot-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) ~[spring-boot-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:124) ~[spring-boot-test-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:123) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.junit.jupiter.SpringExtension.getApplicationContext(SpringExtension.java:202) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.test.context.junit.jupiter.SpringExtension.resolveParameter(SpringExtension.java:188) ~[spring-test-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameter(ExecutableInvoker.java:216) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameters(ExecutableInvoker.java:183) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:74) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeTestClassConstructor(ClassBasedTestDescriptor.java:329) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.instantiateTestClass(ClassBasedTestDescriptor.java:276) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.instantiateTestClass(ClassTestDescriptor.java:77) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.instantiateAndPostProcessTestInstance(ClassBasedTestDescriptor.java:258) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$testInstancesProvider$2(ClassBasedTestDescriptor.java:252) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at java.util.Optional.orElseGet(Optional.java:369) ~[?:?]
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$testInstancesProvider$3(ClassBasedTestDescriptor.java:251) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.execution.TestInstancesProvider.getTestInstances(TestInstancesProvider.java:29) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$prepare$0(TestMethodTestDescriptor.java:106) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.prepare(TestMethodTestDescriptor.java:105) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.prepare(TestMethodTestDescriptor.java:69) ~[junit-jupiter-engine-5.5.2.jar:5.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$prepare$1(NodeTestTask.java:107) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.prepare(NodeTestTask.java:107) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:75) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51) ~[junit-platform-engine-1.5.2.jar:1.5.2]
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220) ~[junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188) ~[junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202) [junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181) [junit-platform-launcher-1.3.1.jar:1.3.1]
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128) [junit-platform-launcher-1.3.1.jar:1.3.1]
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150) [surefire-junit-platform-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124) [surefire-junit-platform-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) [surefire-booter-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) [surefire-booter-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) [surefire-booter-2.22.2.jar:2.22.2]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) [surefire-booter-2.22.2.jar:2.22.2]
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.889 s <<< FAILURE! - in our.group.our.package.CoreApplicationIT
[ERROR] contextLoads Time elapsed: 0.003 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [our.group.our.package.service SomeService someService] in constructor [public our.group.our.package.CoreApplicationIT(our.group.our.package.service.SomeService)]
Caused by: java.lang.ExceptionInInitializerError
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3, imagePullPolicy=DefaultPullPolicy())
Caused by: com.github.dockerjava.api.exception.DockerException:
Status 403: {"message":"authorization denied by plugin pipelines: Invalid content length provided"}
[ERROR] someTest Time elapsed: 0 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [our.group.our.package.service.SomeService someService] in constructor [public our.group.our.package.CoreApplicationIT(our.group.our.package.service.SomeService)]: Could not initialize class our.group.our.package.IntegrationTestBase
Caused by: java.lang.NoClassDefFoundError: Could not initialize class our.group.our.package.IntegrationTestBase
meridsa
All 11 comments
Just snooping on the traffic that goes over the wire when pullImageCmd is used... It looks like neither the okhttp nor httpclient5 transport type clients set a Content-Length header - instead, they use chunked encoding, which is totally allowed per the HTTP spec.
The request headers are like:
POST /images/create?fromImage=alpine&tag=latest HTTP/1.1
accept: application/octet-stream
Accept-Encoding: gzip, x-gzip, deflate
Transfer-Encoding: chunked
Host: localhost:2375
Connection: keep-alive
User-Agent: Apache-HttpClient/5.0 (Java/11.0.3)
Interestingly, looking at the same operation (docker pull) done by the docker CLI:
POST /v1.40/images/create?fromImage=alpine&tag=latest HTTP/1.1
Host: localhost:2375
User-Agent: Docker-Client/19.03.13 (darwin)
Content-Length: 0
Content-Type: text/plain
So it seems that the docker CLI has a preference for setting content length explicitly rather than using chunked encoding.
Based on the error you're seeing, it looks like bitbucket pipelines breaks if chunked encoding is used (i.e. the daemon/daemon auth plugins are not compliant with the http spec). No idea whether this just affects some operations on their daemon or all 馃槩 .
Maybe we could force okhttp and httpclient5 to use an explicit content length? Or ask bitbucket pipelines to look into this?
rnorth
on 1 Oct 2020
I'm stranded on this for the exact same reason as @meridsa. I think this issue is already reported with Bitbucket here: https://jira.atlassian.com/browse/BCLOUD-15844
Doesn't look like it's getting much priority there. I'd make a PR for adding the Content-Length header but I'm new here and I can't really work out where to look for it. Any pointers?
erikhh
on 4 Oct 2020
Hello,
I made a comment on our BCLOUD ticket as to why we dont support transfer encoding chunked Ill link to it here and also replicate its contents.
Unfortunately we cant just add support for Transfer-Encoding: Chunked as the issue itself isnt with the transfer-encoding directly.
We run an authorization plugin attached to every steps docker in docker daemon that inspects the payloads of all the requests to the docker daemon for potentially malicious arguments that would allow users to subvert the security of containers (such as --privileged containers) as we are running a multi tenanted environment we dont want users to be able to subvert this.
How this plugins work is that when the docker daemon receives a request from a client (either the docker-cli, curl, some client in your favourite language etc) it first forwards that request (both the request line e.g. uri and query params and request body) to any authorization plugins which than perform inspections of them and return either an allow or deny (with an optional message) response. see for further information on authorization plugins: https://docs.docker.com/engine/extend/plugins_authorization/
Now unfortunately due to this line of code in the docker daemon https://github.com/moby/moby/blob/master/pkg/authorization/authz.go#L58
if sendBody(ctx.requestURI, r.Header) && r.ContentLength > 0 && r.ContentLength < maxBodySize {
Request bodies are only forwarded to the docker daemon iff they are not for the /auth endpoint and are of content-type: application/json and have a content length that is > 0 and their content length is not > 1mb.
Now unfortunately if your using transfer-encoding: chunked the content-length that most docker clients send through is 0 which is valid for that transfer encoding so that means the request body fails this request and the authorization plugin denies the request as we cant validate your request body doesnt contain any potentially malicious arguments.
Well behaving clients may set the content-length but than there is still and issue in that the body isnt forwarded to the authorization plugin as the docker daemon doesnt first buffer in the request body so if we were to read it it would drain the request body and than the docker daemon would fail the request as it couldnt read the body of the drained stream.
Really docker should improve as this affects not just pipelines but anyone using authorization plugins we are lucky enough to know of this and this can be exploited in other auth plugins who dont perform these extra validations of content-length.
So for now this is the reason why we cant support transfer-encoding: chunked.
I know that your library has multiple backends and one of them defaults to using transfer-encoding: chunked I cant remember which one but (it was either jersey or netty) so users can simply change the backend as a workaround until docker fixes this.
nathanburrell
on 14 Oct 2020
We are having the same problem here. I don't know if by backend you are referring to various transport type (transport.type) in testcontainers.properties but I have tested with each avaialble values and I have the problem anyway.
transport.type=netty -> as fat as I can see, does not work anymore with 1.15.0-rc2.
transport.type=okhttp -> 403
transport.type=httpclient5 -> 403
So it seems we are kind of stuck here and that only alternative for those who use bitbucket pipelines and testcontainers is to downgrade the docker version from 2.4.0 (on mac) to a previous version ?
berkum
on 15 Oct 2020
With all respect, I don't think the problem is with Docker for Mac.
I ran into this issue by upgrading from tescontainers-java 1.14.3 to testcontainers-java 1.15.0-rc2.
Testcontainers-java 1.14.3 has no problem doing it's thing from Bitbucket cloud. Why is it not possible to make it work for 1.15.0? This suggests that everybody that runs their builds in Bitbucket pipelines will have to stick with testcontainers-java 1.14.3 from here on forward.
erikhh
on 16 Oct 2020
@nathanburrell thank you so much for such a detailed comment!
As a workaround, I will investigate adding the content length header to okhttp/ahc5 where possible 馃憤 Although it would be really nice if Docker improves the handling of such requests in the future
bsideup
on 16 Oct 2020
Any status on this? I'm running into the same problem. When docker desktop updated to 1.4.0, I had to upgrade to testcontainers-java 1.15.0-rc2. Now it's breaking builds in bitbucket.
ghost
on 3 Nov 2020
@nathanburrell FYI I just submitted https://github.com/docker-java/docker-java/pull/1495 that makes every non-streaming request sent as a bytearray (meaning - the content-length header will be set). It should solve 99% of requests and unblock the usage of Testcontainers 1.15.x on BitBucket Pipelines. Please let me know if you think that this won't, tho :)
bsideup
on 16 Nov 2020
The workaround was released in 3.2.6. It fixed the problem in our case 馃憤
dagguh
on 24 Nov 2020
@dagguh thanks for trying & reporting! 馃憤
bsideup
on 24 Nov 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 24 Dec 2020
Related issues
fmarchioni
路
3Comments
kpnarayanan
路
4Comments
TomLefley
路
5Comments
priyanka-01
路
4Comments
Sakib37
路
3Comments
Most helpful comment
@nathanburrell thank you so much for such a detailed comment!
As a workaround, I will investigate adding the content length header to okhttp/ahc5 where possible 馃憤 Although it would be really nice if Docker improves the handling of such requests in the future