Jkube: Error processing tar file(archive/tar: missed writing 4096 bytes):

[Sheldor5]

Description

The jkube:build goal failes on CentOS Linux release 7.9.2009 (Core) with:

[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.3.0:build (default) on project myapp-docker: Failed to execute the build: Error while trying to build the image: Unable to build image [someco/myapp:latest] : {"message":"Error processing tar file(archive/tar: missed writing 4096 bytes): "} (Internal Server Error: 500) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.3.0:build (default) on project myapp-docker: Failed to execute the build
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to execute the build
        at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.buildAndTag(AbstractDockerMojo.java:665)
        at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.processImageConfig(AbstractDockerMojo.java:503)
        at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.executeBuildGoal(AbstractDockerMojo.java:632)
        at org.eclipse.jkube.maven.plugin.mojo.build.BuildMojo.executeInternal(BuildMojo.java:49)
        at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.execute(AbstractDockerMojo.java:444)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 20 more
Caused by: org.eclipse.jkube.kit.config.service.JKubeServiceException: Error while trying to build the image: Unable to build image [someco/myapp:latest] : {"message":"Error processing tar file(archive/tar: missed writing 4096 bytes): "} (Internal Server Error: 500)
        at org.eclipse.jkube.kit.config.service.kubernetes.DockerBuildService.build(DockerBuildService.java:52)
        at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.buildAndTag(AbstractDockerMojo.java:662)
        ... 26 more
Caused by: org.eclipse.jkube.kit.build.service.docker.access.DockerAccessException: Unable to build image [someco/myapp:latest] : {"message":"Error processing tar file(archive/tar: missed writing 4096 bytes): "} (Internal Server Error: 500)
        at org.eclipse.jkube.kit.build.service.docker.access.hc.DockerAccessWithHcClient.buildImage(DockerAccessWithHcClient.java:272)
        at org.eclipse.jkube.kit.build.service.docker.BuildService.doBuildImage(BuildService.java:179)
        at org.eclipse.jkube.kit.build.service.docker.BuildService.buildImage(BuildService.java:143)
        at org.eclipse.jkube.kit.build.service.docker.BuildService.buildImage(BuildService.java:77)
        at org.eclipse.jkube.kit.config.service.kubernetes.DockerBuildService.build(DockerBuildService.java:44)
        ... 27 more
Caused by: org.eclipse.jkube.kit.build.service.docker.access.hc.http.HttpRequestException: {"message":"Error processing tar file(archive/tar: missed writing 4096 bytes): "} (Internal Server Error: 500)
        at org.eclipse.jkube.kit.build.service.docker.access.hc.ApacheHttpClientDelegate$StatusCodeCheckerResponseHandler.handleResponse(ApacheHttpClientDelegate.java:200)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:223)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
        at org.eclipse.jkube.kit.build.service.docker.access.hc.ApacheHttpClientDelegate.post(ApacheHttpClientDelegate.java:121)
        at org.eclipse.jkube.kit.build.service.docker.access.hc.DockerAccessWithHcClient.buildImage(DockerAccessWithHcClient.java:270)
        ... 31 more

The same project is working on Windows.

The TAR file looks fine with tar -tf docker-build.tar.

The image can be built manually by going into the assembly directory and running docker build . so docker and the docker daemon are working.

The TAR file is somehow not valid for the docker daemon:

[root@srv3 tmp]# cat docker-build.tar | docker build -
Sending build context to Docker daemon   212 kB
Error response from daemon: Error processing tar file(archive/tar: missed writin 4096 bytes):
[root@srv3 tmp]#

Info

• Eclipse JKube version : 1.3.0
• Maven version (mvn -v) : 3.6.3

[davidecavestro]

We are experiencing something similar.

The build is ok on dev local envs (recent Debian, Ubuntu) but fails on an old Docker 1.12.1 (Git commit: 23cf638) using OpenJDK 1.8.0_111-b15 and Maven 3.3.9 (jkube k8s maven plugin 1.3.0) on CentOS Linux release 7.2.1511

In this case the build fails with a "Connection reset by peer" obscure message, maybe due to the old docker daemon version that internally logs the archjive/tar error message, but - again - manually sending the tar generated by the plugin to the docker daemon we get the very same error

cat target/docker/path/to/tmp/docker-build.tar | docker build -
Error response from daemon: Error processing tar file(archive/tar: missed writing 20 bytes):

even if the content of the tar seems ok

[root@govbuildlx01 jkube]# tar tvf target/docker/path/to/tmp/docker-build.tar
drwxr-xr-x 0/0              20 2021-07-20 14:11 opt/
drwxr-xr-x 0/0              16 2021-07-20 14:11 opt/xyz/
drwxr-xr-x 0/0              21 2021-07-20 14:11 opt/xyz/app/
-rw-r--r-- 0/0             107 2021-07-20 14:11 Dockerfile
-rw-r--r-- 0/0        19337774 2021-07-20 14:11 opt/xyz/app/test.jar

[manusa]

Is there a way to reproduce the issue so we can further investigate?

? Is the result the same if you load->build the tar using:

docker load -i target/docker/path/to/tmp/docker-build.tar

[davidecavestro]

The docker load command fails with

open /var/lib/docker/tmp/docker-import-738275560/opt/json: no such file or directory

both on the problematic server and in dev local env.

At https://github.com/davidecavestro/vagrant-jkube-connection-reset-by-peer I've shared some stuff to reproduce the issue on a vagrant box.

[davidecavestro]

The issue is consistently reproducible from docker-engine 1.12 up to docker-ce 17.12
(with rpms respectively from https://mirrors.aliyun.com/docker-engine/yum/repo/main/centos/7/Packages/ and https://mirrors.aliyun.com/docker-ce/linux/centos/7.2/x86_64/stable/Packages/)

[manusa]

So, do you mean this is specific to the environment (Docker version + OS)? i.e. It does work on new versions of Docker ??

I'm checking your vagrant example (:raised_hands: thanks for this). But I had some trouble running that on Fedora, also I think the vagrant file is not provisioning the project files (checking this now in another machine).

[davidecavestro]

The docker version seems to play a role: switching to a docke versionr >=18.06 in the vagrant box the image is built with no issues.
On my debian and ubuntu I have only recent docker versions, and the issue is not reproducible.

I think the vagrant file is not provisioning the project files

When the box is up and running you could issue a vagrant rsync from your host and check the files in the /vagrant folder within the box, i.e.

dcavestro@lxu-n4705:~/tmp/vagrant/jkube/vagrant-jkube-connection-reset-by-peer$ vagrant rsync
==> default: Rsyncing folder: /home/dcavestro/tmp/vagrant/jkube/vagrant-jkube-connection-reset-by-peer/ => /vagrant
dcavestro@lxu-n4705:~/tmp/vagrant/jkube/vagrant-jkube-connection-reset-by-peer$ vagrant ssh
Last login: Wed Jul 21 00:37:59 2021
[vagrant@localhost ~]$ cd /vagrant
[vagrant@localhost vagrant]$ tree
.
├── pom.xml
├── README.md
├── src
│   └── main
│       └── java
│           └── com
│               └── acme
│                   └── jkube
│                       └── TestApplication.java
└── Vagrantfile

6 directories, 4 files
[vagrant@localhost vagrant]$

[manusa]

When the box is up and running you could issue a vagrant rsync from your host and check the files in the /vagrant folder within the box, i.e

I'm running this on Windows, and it doesn't work very well :(

What I want to try is to generate the docker-build.tar in the vagrant provisioned machine, and then load that tar to the host machine's docker daemon. I think we might be building a Docker archive that's not compatible with those older Docker versions.

There are alternative docker _packagings_, compression available, this is something else I want to try.

[davidecavestro]

Given a certain version of the jkube plugin, mvn and jdk, I guess the tar generated is the same both within the vagrant box or on the host.
So if you could install an "ancient" docker on your host maybe you could reproduce it as well.
Or even on a virtualbox or any other virtualized SO where you send that TAR to an old docker.

I'd say the issue arises from the golang tar library built within the docker binary, i.e. the Flush() function in https://golang.org/src/archive/tar/writer.go
Maybe the tar is not padded as it wants?

[manusa]

Given a certain version of the jkube plugin, mvn and jdk, I guess the tar generated is the same both within the vagrant box or on the host.

Yes, this is what I suspect. But there are other considerations such as how access to the Docker daemon is performed.

Anyway, I'll try something else to reproduce since the Vagrant option is not working very well.

Maybe the tar is not padded as it wants?

I'm not following 100% here. Internally we use commons-compress to generate the docker-build.tar archive (https://github.com/eclipse/jkube/blob/120ee60c1217b1bd24cbfbd4cd49ebf876e5f673/jkube-kit/common/src/main/java/org/eclipse/jkube/kit/common/archive/JKubeTarArchiver.java#L53).
Maybe you can debug this part and see if switching some of the options provides a compliant tar archive.

Is there some specific reason on why you are running older Docker versions? If updating them is not possible, have you tried using jib build strategy as a workaround (-Djkube.build.strategy=jib)?

[davidecavestro]

Never tried with jib build strategy here. Great idea, I'll give it a try.
Updating Docker is the last resort, as this is an old CI server manually mantained by other folks so far, hence potentially plenty of pitfalls.
About commons-compress it would be interesting to compare the tar with one generated by tar cli (maybe I'll give it a try)

[davidecavestro]

About jib strategy, it builds the image, but fails pushing it to the registry (it's an insecure registry sadly exposed only over HTTP)

java.net.SocketException: Network is unreachable (connect failed)

That is the same error I get if I issue a curl to the registry using https, while a curl over http or a docker push work, since the registry is listed as an insecure one in docker daemon config.

Any clue on how to push to an insecure registry using jib? (creds over http)
I've seen https://github.com/eclipse/jkube/issues/336 but it is about pull.

[manusa]

Any clue on how to push to an insecure registry using jib? (creds over http)

:( Could you open a specific feature request issue for that :pray:

[davidecavestro]

I have sketched https://github.com/eclipse/jkube/issues/782

[davidecavestro]

As a side note, the whole thing works perfectly with _fabric8-maven-plugin:3.1.63_.
Maybe that's a clue.

[rohanKanojia]

@davidecavestro : that's quite old version of fabric8-maven-plugin. What about newer versions 4.4.1?

[davidecavestro]

Yep, it works even with 4.4.1.

[manusa]

I think in FMP we relied on Maven to generate the tar file. We really need to see the difference of a tar file generated with one plugin and the other.
With two equivalent configurations (image model), the tar generated from FMP and JKube should be exactly the same.

[rohanKanojia]

@davidecavestro : I'm trying to reproduce your issue. I cloned https://github.com/davidecavestro/vagrant-jkube-connection-reset-by-peer adding fabric8-maven-plugin with the same configuration. But mvn fabric8:build is failing with this error:

[INFO] --- fabric8-maven-plugin:4.4.1:build (default-cli) @ test ---
[INFO] F8: Running in Kubernetes mode
[INFO] F8: Building Container image with Docker in Kubernetes mode
[INFO] Building tar: /home/rokumar/work/repos/vagrant-jkube-connection-reset-by-peer/target/docker/test/1.3.0/tmp/docker-build.tar
[INFO] F8: [test:1.3.0]: Created docker-build.tar in 21 milliseconds
[ERROR] F8: Failed to execute the build: io.fabric8.maven.docker.access.DockerAccessException: Unable to build image [test:1.3.0] : "COPY failed: file not found in build context or excluded by .dockerignore: stat maven: file does not exist" 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.071 s
[INFO] Finished at: 2021-07-23T15:17:22+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:fabric8-maven-plugin:4.4.1:build (default-cli) on project test: Failed to execute the build: io.fabric8.maven.docker.access.DockerAccessException: Unable to build image [test:1.3.0] : "COPY failed: file not found in build context or excluded by .dockerignore: stat maven: file does not exist"  -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

When you said it's working with 4.4.1, did you mean this reproducer project or some other project?

[davidecavestro]

Yep, I meant the reproducer project, with the assembly portion modified to

<assembly>
    <descriptorRef>artifact</descriptorRef>
    <basedir>/opt/xyz/app</basedir>
</assembly>

[davidecavestro]

This works with FMP, but introduces changes into the generated Dockerfile and how the temporary structure is packaged into the tar, hence the two tar don't share the same contents.
Maybe you know how to produce easily comparable tars... but it that case I think you can build/compare them in any environment.

OTOH the vagrant box is meant to give you an old docker that is not able to process the tar from jkube (but accepts the one from fabric8).

[rohanKanojia]

@davidecavestro : Thanks a lot for sharing vagrantfile, I can reproduce your issue

[rohanKanojia]

I noticed that in case of jkube directory size was more than zero:

[vagrant@localhost vagrant]$ tar -tvf /tmp/jkube.tar 
drwxr-xr-x 0/0              22 2021-07-27 10:47 maven/
-rw-r--r-- 0/0              94 2021-07-27 10:47 Dockerfile
-rw-r--r-- 0/0        19337676 2021-07-27 10:47 maven/test.jar
[vagrant@localhost vagrant]$ tar -tvf /tmp/fabric8.tar 
drwxrwxr-x vagrant/vagrant   0 2021-07-27 11:28 maven/
-rw-rw-r-- vagrant/vagrant 19337676 2021-07-27 11:27 maven/test.jar
-rw-rw-r-- vagrant/vagrant       93 2021-07-27 11:28 Dockerfile

When I set TarArchiveEntry size only in case of files, I'm able to get jkube working with docker:
https://github.com/eclipse/jkube/blob/15bb59fbe2eb8c5f26e450ec7e4256bae19fe070/jkube-kit/common/src/main/java/org/eclipse/jkube/kit/common/archive/JKubeTarArchiver.java#L78

[vagrant@localhost vagrant-jkube-connection-reset-by-peer]$ mvn k8s:build
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Dockerfile :: Simple 1.3.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.4.0-SNAPSHOT:build (default-cli) @ test ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: [test:1.3.0]: Created docker-build.tar in 160 milliseconds
[INFO] k8s: [test:1.3.0]: Built image sha256:d4fb4
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.483 s
[INFO] Finished at: 2021-07-27T14:23:10+02:00
[INFO] Final Memory: 31M/273M
[INFO] ------------------------------------------------------------------------
[vagrant@localhost vagrant-jkube-connection-reset-by-peer]$ docker run d4fb4

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[manusa]

For future reference, also note that user/group id, name, etc. are not set in the TarArchiveEntry entities.

https://github.com/codehaus-plexus/plexus-archiver/blob/ad3f2f324271f9303be0d5511e2a9cab5130a840/src/main/java/org/codehaus/plexus/archiver/tar/TarArchiver.java#L296-L313

[rohanKanojia]

Let me create a separate issue for this.

[manusa]

Let me create a separate issue for this.

If it's not necessary I'd rather keep it this way. Maybe the unnecessary fix breaks something else. I just added the comment for the record in case we hit some other issue in the future.

[davidecavestro]

So I've tried the vagrant box sample with 1.4.0. It build the image, but as soon as I add the deploy step just to simulate real-world scenario i.e.
mvn -DskipTests clean package k8s:build deploy -DaltDeploymentRepository=snapshot-repo::default::file:/tmp/foo
I got

[ERROR] Caused by: java.lang.IllegalArgumentException: Can not set org.eclipse.aether.spi.log.Logger field org.apache.maven.repository.internal.DefaultVersionResolver.logger to org.eclipse.aether.internal.impl.slf4j.Slf4jLoggerFactory
[ERROR] at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
[ERROR] at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
[ERROR] at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
[ERROR] at java.lang.reflect.Field.set(Field.java:764)
[ERROR] at org.eclipse.sisu.bean.BeanPropertyField.set(BeanPropertyField.java:72)

I guess it depends on the ancient version of Maven

[rohanKanojia]

@davidecavestro Which version of maven are you using? I saw a similar issue on docker maven plugin today(https://github.com/fabric8io/docker-maven-plugin/issues/1485). Does splitting phases like mentioned in the issue help you use case?

[davidecavestro]

Using mvn 3.3.9.
Splitting phases is not enough if I still call the goals in the following sequence
mvn ... clean package k8s:build deploy

Instead it works fine - splitting phases or not doesn't matter - if I simply call them separately, i.e.

1. mvn ... clean package k8s:build
2. mvn ... deploy

That said the _logging issue_ is not reproducible using a slightly more recent version of maven such as 3.6.3 (trioed that as it is already in use on other CI systems)

[davidecavestro]

(thank you guys and keep up the good work)

[rohanKanojia]

@davidecavestro : Thanks a lot to you for coordinating with us. Your vagrant setup helped a lot in debugging this issue.

[manusa]

Instead it works fine - splitting phases or not doesn't matter - if I simply call them separately

:relieved: At least we got it working (with some extra hassle). Thanks for sharing the workaround.