Jib: Cannot use base image hosted on ghcr

[anuraaga]

Environment:

• Jib version: 2.6.0
• Build tool: Gradle
• OS: Linux

Description of the issue:
When using Jib to build an image with a base image hosted on GitHub Container Registry (ghcr.io), it fails with Premature end of Content-Length delimited message body

Expected behavior:

Build should succeed with ghcr.io base image

Steps to reproduce:

1. git clone https://github.com/anuraaga/jib-ghcr.git
2. cd jib-ghcr
3. ./gradlew jibDockerBuild

jib-gradle-plugin Configuration:

jib {
  to {
    image = "ghcr.io/anuraaga/foobar"
  }
  from {
    image = "ghcr.io/github/super-linter:v3"
  }
  container {
    mainClass = "Application"
  }
}

I have tried with other ghcr images (I originally tried with my own image and switched to github's example super-linter)

Log output:

Base image 'ghcr.io/github/super-linter:v3' does not use a specific image digest - build may not be reproducible
The base image requires auth. Trying again for ghcr.io/github/super-linter:v3...
Using credentials from Docker config (/Users/aanuraag/.docker/config.json) for ghcr.io/github/super-linter:v3
Using base image with digest: sha256:b5b43c0f80224919968b1482b48d82c2424cf596e09468917025575a6bf110d2
I/O error for image [ghcr.io/github/super-linter]:
    org.apache.http.ConnectionClosedException
    Premature end of Content-Length delimited message body (expected: 31,167; received: 0)

Additional Information:

The base image works fine with docker pull or building a Dockerfile. Jib works fine with images hosted on other container registries. ghcr base images used to work, but stopped working last week or so, presumably due to a change in ghcr. However, since the docker command works fine, presumably it's something that can be fixed in jib too. Perhaps it's time to swap apache HTTP with okhttp.

Also just to make sure it's not an auth related issue I tried with removing .docker/config.json and same error.

[chanseokoh]

Thanks for the report.

The root cause is that the server sends a non-zero Content-Length: HTTP header for a 307 Temporary Redirect response. It means the server is telling the client that it's attaching a content body of size 31167 in the response, which of course isn't the case as a redirect response.

HTTP/1.1 307 Temporary Redirect
Content-Length: 31167
Content-Type: application/octet-stream
Docker-Content-Digest: sha256:c7ac13e5f90fc4ec084e37323433806d2b693ec0954243b5fd1eac938db3c1b7
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:c7ac13e5f90fc4ec084e37323433806d2b693ec0954243b5fd1eac938db3c1b7"
Location: https://github-production-container-registry.s3.amazonaws.com/blobs/sha256%3Ac7ac13e5f90fc4ec084e37323433806d2b693ec0954243b5fd1eac938db3c1b7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYXOCC6LC2SUN5FPM%2F20201019%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201019T143034Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=b4603c19d939e65e7eb93294ad854095dbe5173daffd2f3344db2eae30ece915
Date: Mon, 19 Oct 2020 14:30:34 GMT
X-GitHub-Request-Id: CDB2:4010:722EA:118BF2:5F8DA309

I suspected that this is a server bug, but to confirm that the server is not compliant to the HTTP spec, I filed a bug against Apache (HTTPCLIENT-2121). They say that the server must return Content-Length: 0 in this case.

Sending Content-Length: 0 in redirect response is an absolute MUST for minimal protocol compliance.

And I agree GHCR should fix it. Would you be willing to file a bug against GCHR? We'd appreciate your update here as well.

[anuraaga]

And I agree GHCR should fix it.

@chanseokoh Well technically not agreeing :) I figured if this works with the normal docker client (presumably just the normal Go HTTP library) it's a good idea for jib to support it for these misbehaving servers for parity with docker? I'll try to follow up with ghcr too though.

[chanseokoh]

I see your point, but it's tricky to fix this on the Jib side, as the error comes from the Apache HttpClient handling the protocol at a far lower level. And seems like it's unlikely that the Apache folks are willing to sidestep this. (Generally they maintain the purest point of view and are seriously against tolerating misbehaving servers.) GHCR is public beta now, and hopefully it's an easy fix. Have you filed a bug, issue, or whatever? Any link?

[anuraaga]

Thanks - yeah I filed a support ticket with ghcr and got a response. They appreciated the detail you provided :) I don't think it's a public link unfortunately, will keep you updated.

[clarkbw]

👋 we rolled out a fix for this header about 2 hours ago, you should see it working now. Thanks! 🙇

[anuraaga]

@clarkbw Thanks for the quick fix! Confirmed it's working fine now :)